diff options
Diffstat (limited to 'tests/test-runner/man')
| -rw-r--r-- | tests/test-runner/man/Makefile.am | 4 | ||||
| -rw-r--r-- | tests/test-runner/man/test-runner.1 | 370 |
2 files changed, 374 insertions, 0 deletions
diff --git a/tests/test-runner/man/Makefile.am b/tests/test-runner/man/Makefile.am new file mode 100644 index 000000000..a7017f5f0 --- /dev/null +++ b/tests/test-runner/man/Makefile.am @@ -0,0 +1,4 @@ +dist_man_MANS = test-runner.1 + +install-data-local: + $(INSTALL) -d -m 0755 "$(DESTDIR)$(mandir)/man1" diff --git a/tests/test-runner/man/test-runner.1 b/tests/test-runner/man/test-runner.1 new file mode 100644 index 000000000..31cd41245 --- /dev/null +++ b/tests/test-runner/man/test-runner.1 @@ -0,0 +1,370 @@ +.\" +.\" This file and its contents are supplied under the terms of the +.\" Common Development and Distribution License ("CDDL"), version 1.0. +.\" You may only use this file in accordance with the terms of version +.\" 1.0 of the CDDL. +.\" +.\" A full copy of the text of the CDDL should have accompanied this +.\" source. A copy of the CDDL is also available via the Internet at +.\" http://www.illumos.org/license/CDDL. +.\" +.\" +.\" Copyright (c) 2012 by Delphix. All rights reserved. +.\" +.TH run 1 "23 Sep 2012" +.SH NAME +run \- find, execute, and log the results of tests +.SH SYNOPSIS +.LP +.nf +\fBrun\fR [\fB-dgq] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR] [\fB-uxX\fR \fIusername\fR] + \fIpathname\fR ... +.fi + +.LP +.nf +\fBrun\fR \fB-w\fR \fIrunfile\fR [\fB-gq\fR] [\fB-o\fR \fIoutputdir\fR] [\fB-pP\fR \fIscript\fR] [\fB-t\fR \fIseconds\fR] + [\fB-uxX\fR \fIusername\fR] \fIpathname\fR ... +.fi + +.LP +.nf +\fBrun\fR \fB-c\fR \fIrunfile\fR [\fB-dq\fR] +.fi + +.LP +.nf +\fBrun\fR [\fB-h\fR] +.fi + +.SH DESCRIPTION +.sp +.LP +The \fBrun\fR command has three basic modes of operation. With neither the +\fB-c\fR nor the \fB-w\fR option, \fBrun\fR processes the arguments provided on +the command line, adding them to the list for this run. If a specified +\fIpathname\fR is an executable file, it is added as a test. If a specified +\fIpathname\fR is a directory, the behavior depends upon the \fB-g\fR option. +If \fB-g\fR is specified, the directory is treated as a test group. See the +section on "Test Groups" below. Without the \fB-g\fR option, \fBrun\fR simply +descends into the directory looking for executable files. The tests are then +executed, and the results are logged. + +With the \fB-w\fR option, \fBrun\fR finds tests in the manner described above. +Rather than executing the tests and logging the results, the test configuration +is stored in a \fIrunfile\fR which can be used in future invocations, or edited +to modify which tests are executed and which options are applied. Options +included on the command line with \fB-w\fR become defaults in the +\fIrunfile\fR. + +With the \fB-c\fR option, \fBrun\fR parses a \fIrunfile\fR, which can specify a +series of tests and test groups to be executed. The tests are then executed, +and the results are logged. +.sp +.SS "Test Groups" +.sp +.LP +A test group is comprised of a set of executable files, all of which exist in +one directory. The options specified on the command line or in a \fIrunfile\fR +apply to individual tests in the group. The exception is options pertaining to +pre and post scripts, which act on all tests as a group. Rather than running +before and after each test, these scripts are run only once each at the start +and end of the test group. +.SS "Test Execution" +.sp +.LP +The specified tests run serially, and are typically assigned results according +to exit values. Tests that exit zero and non-zero are marked "PASS" and "FAIL" +respectively. When a pre script fails for a test group, only the post script is +executed, and the remaining tests are marked "SKIPPED." Any test that exceeds +its \fItimeout\fR is terminated, and marked "KILLED." + +By default, tests are executed with the credentials of the \fBrun\fR script. +Executing tests with other credentials is done via \fBsudo\fR(1m), which must +be configured to allow execution without prompting for a password. Environment +variables from the calling shell are available to individual tests. During test +execution, the working directory is changed to \fIoutputdir\fR. +.SS "Output Logging" +.sp +.LP +By default, \fBrun\fR will print one line on standard output at the conclusion +of each test indicating the test name, result and elapsed time. Additionally, +for each invocation of \fBrun\fR, a directory is created using the ISO 8601 +date format. Within this directory is a file named \fIlog\fR containing all the +test output with timestamps, and a directory for each test. Within the test +directories, there is one file each for standard output, standard error and +merged output. The default location for the \fIoutputdir\fR is +\fI/var/tmp/test_results\fR. +.SS "Runfiles" +.sp +.LP +The \fIrunfile\fR is an ini style configuration file that describes a test run. +The file has one section named "DEFAULT," which contains configuration option +names and their values in "name = value" format. The values in this section +apply to all the subsequent sections, unless they are also specified there, in +which case the default is overridden. The remaining section names are the +absolute pathnames of files and direcotries, describing tests and test groups +respectively. The legal option names are: +.sp +.ne 2 +.na +\fBoutputdir\fR = \fIpathname\fR +.ad +.sp .6 +.RS 4n +The name of the directory that holds test logs. +.RE +.sp +.ne 2 +.na +\fBpre\fR = \fIscript\fR +.ad +.sp .6 +.RS 4n +Run \fIscript\fR prior to the test or test group. +.RE +.sp +.ne 2 +.na +\fBpre_user\fR = \fIusername\fR +.ad +.sp .6 +.RS 4n +Execute the pre script as \fIusername\fR. +.RE +.sp +.ne 2 +.na +\fBpost\fR = \fIscript\fR +.ad +.sp .6 +.RS 4n +Run \fIscript\fR after the test or test group. +.RE +.sp +.ne 2 +.na +\fBpost_user\fR = \fIusername\fR +.ad +.sp .6 +.RS 4n +Execute the post script as \fIusername\fR. +.RE +.sp +.ne 2 +.na +\fBquiet\fR = [\fITrue\fR|\fIFalse\fR] +.ad +.sp .6 +.RS 4n +If set to True, only the results summary is printed to standard out. +.RE +.sp +.ne 2 +.na +\fBtests\fR = [\fI'filename'\fR [,...]] +.ad +.sp .6 +.RS 4n +Specify a list of \fIfilenames\fR for this test group. Only the basename of the +absolute path is required. This option is only valid for test groups, and each +\fIfilename\fR must be single quoted. +.RE +.sp +.ne 2 +.na +\fBtimeout\fR = \fIn\fR +.ad +.sp .6 +.RS 4n +A timeout value of \fIn\fR seconds. +.RE +.sp +.ne 2 +.na +\fBuser\fR = \fIusername\fR +.ad +.sp .6 +.RS 4n +Execute the test or test group as \fIusername\fR. +.RE + +.SH OPTIONS +.sp +.LP +The following options are available for the \fBrun\fR command. +.sp +.ne 2 +.na +\fB-c\fR \fIrunfile\fR +.ad +.RS 6n +Specify a \fIrunfile\fR to be consumed by the run command. +.RE + +.ne 2 +.na +\fB-d\fR +.ad +.RS 6n +Dry run mode. Execute no tests, but print a description of each test that would +have been run. +.RE + +.ne 2 +.na +\fB-g\fR +.ad +.RS 6n +Create test groups from any directories found while searching for tests. +.RE + +.ne 2 +.na +\fB-o\fR \fIoutputdir\fR +.ad +.RS 6n +Specify the directory in which to write test results. +.RE + +.ne 2 +.na +\fB-p\fR \fIscript\fR +.ad +.RS 6n +Run \fIscript\fR prior to any test or test group. +.RE + +.ne 2 +.na +\fB-P\fR \fIscript\fR +.ad +.RS 6n +Run \fIscript\fR after any test or test group. +.RE + +.ne 2 +.na +\fB-q\fR +.ad +.RS 6n +Print only the results sumary to the standard output. +.RE + +.ne 2 +.na +\fB-t\fR \fIn\fR +.ad +.RS 6n +Specify a timeout value of \fIn\fR seconds per test. +.RE + +.ne 2 +.na +\fB-u\fR \fIusername\fR +.ad +.RS 6n +Execute tests or test groups as \fIusername\fR. +.RE + +.ne 2 +.na +\fB-w\fR \fIrunfile\fR +.ad +.RS 6n +Specify the name of the \fIrunfile\fR to create. +.RE + +.ne 2 +.na +\fB-x\fR \fIusername\fR +.ad +.RS 6n +Execute the pre script as \fIusername\fR. +.RE + +.ne 2 +.na +\fB-X\fR \fIusername\fR +.ad +.RS 6n +Execute the post script as \fIusername\fR. +.RE + +.SH EXAMPLES +.LP +\fBExample 1\fR Running ad-hoc tests. +.sp +.LP +This example demonstrates the simplest invocation of \fBrun\fR. + +.sp +.in +2 +.nf +% \fBrun my-tests\fR +Test: /home/jkennedy/my-tests/test-01 [00:02] [PASS] +Test: /home/jkennedy/my-tests/test-02 [00:04] [PASS] +Test: /home/jkennedy/my-tests/test-03 [00:01] [PASS] + +Results Summary +PASS 3 + +Running Time: 00:00:07 +Percent passed: 100.0% +Log directory: /var/tmp/test_results/20120923T180654 +.fi +.in -2 + +.LP +\fBExample 2\fR Creating a \fIrunfile\fR for future use. +.sp +.LP +This example demonstrates creating a \fIrunfile\fR with non default options. + +.sp +.in +2 +.nf +% \fBrun -p setup -x root -g -w new-tests.run new-tests\fR +% \fBcat new-tests.run\fR +[DEFAULT] +pre = setup +post_user = +quiet = False +user = +timeout = 60 +post = +pre_user = root +outputdir = /var/tmp/test_results + +[/home/jkennedy/new-tests] +tests = ['test-01', 'test-02', 'test-03'] +.fi +.in -2 + +.SH EXIT STATUS +.sp +.LP +The following exit values are returned: +.sp +.ne 2 +.na +\fB\fB0\fR\fR +.ad +.sp .6 +.RS 4n +Successful completion. +.RE +.sp +.ne 2 +.na +\fB\fB1\fR\fR +.ad +.sp .6 +.RS 4n +An error occurred. +.RE + +.SH SEE ALSO +.sp +.LP +\fBsudo\fR(1m) |