Archive-Team
/
zfs
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 103 Wiki Activity

test-runner.1: modernise 

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #12125
This commit is contained in:
наб 2021-05-26 20:15:21 +02:00 committed by Brian Behlendorf
parent 71f6e6480e
commit f2e890ddfa
1 changed files with 222 additions and 325 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

547
tests/test-runner/man/test-runner.1
View File

@ -11,313 +11,241 @@
.\" .\"
.\" Copyright (c) 2012 by Delphix. All rights reserved. .\" Copyright (c) 2012 by Delphix. All rights reserved.
.\" .\"
.TH run 1 "23 Sep 2012" .Dd March 10, 2020
.SH NAME .Dt RUN 1
run \- find, execute, and log the results of tests .Os
.SH SYNOPSIS .
.LP .Sh NAME
.nf .Nm run
\fBrun\fR [\fB-dgq] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR] [\fB-uxX\fR \fIusername\fR] .Nd find, execute, and log the results of tests
\fIpathname\fR ... .Sh SYNOPSIS
.fi .Nm
.Op Fl dgq
.LP .Op Fl o Ar outputdir
.nf .Op Fl pP Ar script
\fBrun\fR \fB-w\fR \fIrunfile\fR [\fB-gq\fR] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR] .Op Fl t seconds
[\fB-uxX\fR \fIusername\fR] \fIpathname\fR ... .Op Fl uxX Ar username
.fi .Ar pathname Ns No …
.Pp
.LP .Nm
.nf .Fl w Ar runfile
\fBrun\fR \fB-c\fR \fIrunfile\fR [\fB-dq\fR] .Op Fl gq
.fi .Op Fl o Ar outputdir
.Op Fl pP Ar script
.LP .Op Fl t seconds
.nf .Op Fl uxX Ar username
\fBrun\fR [\fB-h\fR] .Ar pathname Ns No …
.fi .Pp
.Nm
.SH DESCRIPTION .Fl c Ar runfile
.sp .Op Fl dq
.LP .Pp
The \fBrun\fR command has three basic modes of operation. With neither the .Nm
\fB-c\fR nor the \fB-w\fR option, \fBrun\fR processes the arguments provided on .Op Fl h
.
.Sh DESCRIPTION
.Nm
command has three basic modes of operation. With neither
.Fl c
nor
.Fl w ,
.Nm
processes the arguments provided on
the command line, adding them to the list for this run. If a specified the command line, adding them to the list for this run. If a specified
\fIpathname\fR is an executable file, it is added as a test. If a specified .Ar pathname
\fIpathname\fR is a directory, the behavior depends upon the \fB-g\fR option. is an executable file, it is added as a test. If a specified
If \fB-g\fR is specified, the directory is treated as a test group. See the .Ar pathname
section on "Test Groups" below. Without the \fB-g\fR option, \fBrun\fR simply is a directory, the behavior depends upon the presence of
descends into the directory looking for executable files. The tests are then .Fl g .
executed, and the results are logged. If
.Fl g
With the \fB-w\fR option, \fBrun\fR finds tests in the manner described above. is specified, the directory is treated as a test group. See the
section on
.Sy Test Groups
below. Without
.Fl g ,
.Nm
simply descends into the directory looking for executable files.
The tests are then executed, and the results are logged.
.Pp
With
.Fl w ,
.Nm
finds tests in the manner described above.
Rather than executing the tests and logging the results, the test configuration Rather than executing the tests and logging the results, the test configuration
is stored in a \fIrunfile\fR which can be used in future invocations, or edited is stored in a
.Ar runfile ,
which can be used in future invocations, or edited
to modify which tests are executed and which options are applied. Options to modify which tests are executed and which options are applied. Options
included on the command line with \fB-w\fR become defaults in the included on the command line with
\fIrunfile\fR. .Fl w
become defaults in the
With the \fB-c\fR option, \fBrun\fR parses a \fIrunfile\fR, which can specify a .Ar runfile .
series of tests and test groups to be executed. The tests are then executed, .Pp
and the results are logged. With
.sp .Fl c ,
.SS "Test Groups" .Nm
.sp parses a
.LP .Ar runfile ,
which can specify a series of tests and test groups to be executed.
The tests are then executed, and the results are logged.
.
.Ss Test Groups
A test group is comprised of a set of executable files, all of which exist in A test group is comprised of a set of executable files, all of which exist in
one directory. The options specified on the command line or in a \fIrunfile\fR one directory. The options specified on the command line or in a
.Ar runfile
apply to individual tests in the group. The exception is options pertaining to apply to individual tests in the group. The exception is options pertaining to
pre and post scripts, which act on all tests as a group. Rather than running pre and post scripts, which act on all tests as a group. Rather than running
before and after each test, these scripts are run only once each at the start before and after each test, these scripts are run only once each at the start
and end of the test group. and end of the test group.
.SS "Test Execution" .Ss Test Execution
.sp
.LP
The specified tests run serially, and are typically assigned results according The specified tests run serially, and are typically assigned results according
to exit values. Tests that exit zero and non-zero are marked "PASS" and "FAIL" to exit values. Tests that exit zero and non-zero are marked
.Sy PASS
and
.Sy FAIL ,
respectively. When a pre script fails for a test group, only the post script is respectively. When a pre script fails for a test group, only the post script is
executed, and the remaining tests are marked "SKIPPED." Any test that exceeds executed, and the remaining tests are marked
its \fItimeout\fR is terminated, and marked "KILLED." .Sy SKIPPED .
Any test that exceeds
By default, tests are executed with the credentials of the \fBrun\fR script. its
Executing tests with other credentials is done via \fBsudo\fR(1m), which must .Ar timeout
is terminated, and marked
.Sy KILLED .
.Pp
By default, tests are executed with the credentials of the
.Sy
script.
Executing tests with other credentials is done via
.Xr sudo 1m ,
which must
be configured to allow execution without prompting for a password. Environment be configured to allow execution without prompting for a password. Environment
variables from the calling shell are available to individual tests. During test variables from the calling shell are available to individual tests. During test
execution, the working directory is changed to \fIoutputdir\fR. execution, the working directory is changed to
.SS "Output Logging" .Ar outputdir .
.sp .
.LP .Ss Output Logging
By default, \fBrun\fR will print one line on standard output at the conclusion By default,
.Nm
will print one line on standard output at the conclusion
of each test indicating the test name, result and elapsed time. Additionally, of each test indicating the test name, result and elapsed time. Additionally,
for each invocation of \fBrun\fR, a directory is created using the ISO 8601 for each invocation of
date format. Within this directory is a file named \fIlog\fR containing all the .Nm ,
a directory is created using the ISO 8601
date format. Within this directory is a file named
.Sy log
containing all the
test output with timestamps, and a directory for each test. Within the test test output with timestamps, and a directory for each test. Within the test
directories, there is one file each for standard output, standard error and directories, there is one file each for standard output, standard error and
merged output. The default location for the \fIoutputdir\fR is merged output. The default location for the
\fI/var/tmp/test_results\fR. .Ar outputdir
.SS "Runfiles" is
.sp .Pa /var/tmp/test_results .
.LP .Ss "Runfiles"
The \fIrunfile\fR is an ini style configuration file that describes a test run. The
The file has one section named "DEFAULT," which contains configuration option .Ar runfile
names and their values in "name = value" format. The values in this section is an INI-style configuration file that describes a test run.
The file has one section named
.Sy DEFAULT ,
which contains configuration option
names and their values in
.Sy name No = Ar value
format. The values in this section
apply to all the subsequent sections, unless they are also specified there, in apply to all the subsequent sections, unless they are also specified there, in
which case the default is overridden. The remaining section names are the which case the default is overridden. The remaining section names are the
absolute pathnames of files and directories, describing tests and test groups absolute pathnames of files and directories, describing tests and test groups
respectively. The legal option names are: respectively. The legal option names are:
.sp .Bl -tag -width "tests = ['filename', …]"
.ne 2 .It Sy outputdir No = Ar pathname
.na
\fBoutputdir\fR = \fIpathname\fR
.ad
.sp .6
.RS 4n
The name of the directory that holds test logs. The name of the directory that holds test logs.
.RE .It Sy pre No = Ar script
.sp Run
.ne 2 .Ar script
.na prior to the test or test group.
\fBpre\fR = \fIscript\fR .It Sy pre_user No = Ar username
.ad Execute the pre script as
.sp .6 .Ar username .
.RS 4n .It Sy post No = Ar script
Run \fIscript\fR prior to the test or test group. Run
.RE .Ar script
.sp after the test or test group.
.ne 2 .It Sy post_user No = Ar username
.na Execute the post script as
\fBpre_user\fR = \fIusername\fR .Ar username .
.ad .It Sy quiet No = Sy True Ns | Ns Sy False
.sp .6 If
.RS 4n .Sy True ,
Execute the pre script as \fIusername\fR. only the results summary is printed to standard out.
.RE .It Sy tests No = [ Ns Ar 'filename' Ns , …]
.sp Specify a list of
.ne 2 .Ar filenames
.na for this test group. Only the basename of the
\fBpost\fR = \fIscript\fR
.ad
.sp .6
.RS 4n
Run \fIscript\fR after the test or test group.
.RE
.sp
.ne 2
.na
\fBpost_user\fR = \fIusername\fR
.ad
.sp .6
.RS 4n
Execute the post script as \fIusername\fR.
.RE
.sp
.ne 2
.na
\fBquiet\fR = [\fITrue\fR|\fIFalse\fR]
.ad
.sp .6
.RS 4n
If set to True, only the results summary is printed to standard out.
.RE
.sp
.ne 2
.na
\fBtests\fR = [\fI'filename'\fR [,...]]
.ad
.sp .6
.RS 4n
Specify a list of \fIfilenames\fR for this test group. Only the basename of the
absolute path is required. This option is only valid for test groups, and each absolute path is required. This option is only valid for test groups, and each
\fIfilename\fR must be single quoted. .Ar filename
.RE must be single quoted.
.sp .It Sy timeout No = Ar n
.ne 2 A timeout value of
.na .Ar n
\fBtimeout\fR = \fIn\fR seconds.
.ad .It Sy user No = Ar username
.sp .6 Execute the test or test group as
.RS 4n .Ar username .
A timeout value of \fIn\fR seconds. .El
.RE .
.sp .Sh OPTIONS
.ne 2 .Bl -tag -width "-o outputdir"
.na .It Fl c Ar runfile
\fBuser\fR = \fIusername\fR Specify a
.ad .Ar runfile
.sp .6 to be consumed by the run command.
.RS 4n .It Fl d
Execute the test or test group as \fIusername\fR.
.RE
.SH OPTIONS
.sp
.LP
The following options are available for the \fBrun\fR command.
.sp
.ne 2
.na
\fB-c\fR \fIrunfile\fR
.ad
.RS 6n
Specify a \fIrunfile\fR to be consumed by the run command.
.RE
.ne 2
.na
\fB-d\fR
.ad
.RS 6n
Dry run mode. Execute no tests, but print a description of each test that would Dry run mode. Execute no tests, but print a description of each test that would
have been run. have been run.
.RE .It Fl g
.ne 2
.na
\fB-g\fR
.ad
.RS 6n
Create test groups from any directories found while searching for tests. Create test groups from any directories found while searching for tests.
.RE .It Fl o Ar outputdir
.ne 2
.na
\fB-o\fR \fIoutputdir\fR
.ad
.RS 6n
Specify the directory in which to write test results. Specify the directory in which to write test results.
.RE .It Fl p Ar script
Run
.ne 2 .Ar script
.na prior to any test or test group.
\fB-p\fR \fIscript\fR .It Fl P Ar script
.ad Run
.RS 6n .Ar script
Run \fIscript\fR prior to any test or test group. after any test or test group.
.RE .It Fl q
.ne 2
.na
\fB-P\fR \fIscript\fR
.ad
.RS 6n
Run \fIscript\fR after any test or test group.
.RE
.ne 2
.na
\fB-q\fR
.ad
.RS 6n
Print only the results summary to the standard output. Print only the results summary to the standard output.
.RE .It Fl s Ar script
Run
.ne 2 .Ar script as a failsafe after any test is killed.
.na .It Fl S Ar username
\fB-s\fR \fIscript\fR Execute the failsafe script as
.ad .Ar username .
.RS 6n .It Fl t Ar n
Run \fIscript\fR as a failsafe after any test is killed. Specify a timeout value of
.RE .Ar n seconds per test.
.It Fl u Ar username
.ne 2 Execute tests or test groups as
.na .Ar username .
\fB-S\fR \fIusername\fR .It Fl w Ar runfile
.ad Specify the name of the
.RS 6n .Ar runfile
Execute the failsafe script as \fIusername\fR. to create.
.RE .It Fl x Ar username
Execute the pre script as
.ne 2 .Ar username .
.na .It Fl X Ar username
\fB-t\fR \fIn\fR Execute the post script as
.ad .Ar username .
.RS 6n .El
Specify a timeout value of \fIn\fR seconds per test. .
.RE .Sh EXAMPLES
.Bl -tag -width "-h"
.ne 2 .It Sy Example 1 Ns : Running ad-hoc tests.
.na This example demonstrates the simplest invocation of
\fB-u\fR \fIusername\fR .Nm .
.ad .Bd -literal
.RS 6n .No % Nm run Ar my-tests
Execute tests or test groups as \fIusername\fR.
.RE
.ne 2
.na
\fB-w\fR \fIrunfile\fR
.ad
.RS 6n
Specify the name of the \fIrunfile\fR to create.
.RE
.ne 2
.na
\fB-x\fR \fIusername\fR
.ad
.RS 6n
Execute the pre script as \fIusername\fR.
.RE
.ne 2
.na
\fB-X\fR \fIusername\fR
.ad
.RS 6n
Execute the post script as \fIusername\fR.
.RE
.SH EXAMPLES
.LP
\fBExample 1\fR Running ad-hoc tests.
.sp
.LP
This example demonstrates the simplest invocation of \fBrun\fR.
.sp
.in +2
.nf
% \fBrun my-tests\fR
Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS] Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS]
Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS] Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS]
Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS] Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS]
@ -328,20 +256,14 @@ PASS 3
Running Time: 00:00:07 Running Time: 00:00:07
Percent passed: 100.0% Percent passed: 100.0%
Log directory: /var/tmp/test_results/20120923T180654 Log directory: /var/tmp/test_results/20120923T180654
.fi .Ed
.in -2 .It Sy Example 2 Ns : Creating a Ar runfile No for future use.
This example demonstrates creating a
.LP .Ar runfile
\fBExample 2\fR Creating a \fIrunfile\fR for future use. with non-default options.
.sp .Bd -literal
.LP .No % Nm run Fl p Ar setup Fl x Ar root Fl g Fl w Ar new-tests.run Ar new-tests
This example demonstrates creating a \fIrunfile\fR with non default options. .No % Nm cat Pa new-tests.run
.sp
.in +2
.nf
% \fBrun -p setup -x root -g -w new-tests.run new-tests\fR
% \fBcat new-tests.run\fR
[DEFAULT] [DEFAULT]
pre = setup pre = setup
post_user = post_user =
@ -354,33 +276,8 @@ outputdir = /var/tmp/test_results
[/home/jkennedy/new-tests] [/home/jkennedy/new-tests]
tests = ['test-01', 'test-02', 'test-03'] tests = ['test-01', 'test-02', 'test-03']
.fi .Ed
.in -2 .El
.
.SH EXIT STATUS .Sh SEE ALSO
.sp .Xr sudo 1m
.LP
The following exit values are returned:
.sp
.ne 2
.na
\fB\fB0\fR\fR
.ad
.sp .6
.RS 4n
Successful completion.
.RE
.sp
.ne 2
.na
\fB\fB1\fR\fR
.ad
.sp .6
.RS 4n
An error occurred.
.RE
.SH SEE ALSO
.sp
.LP
\fBsudo\fR(1m)