TestRunner

Understanding TestRunner

testrunner

A test runner is a library or tool that picks up an assembly (or a source code directory). In addition, this contains unit tests and a bunch of settings. Subsequently, it executes them and writes the test results to the console or log files.

TestRunner Command-Line Arguments

To run functional tests from a command line. Certainly, you use the testrunner.bat/.sh file that is located in the <SoapUI>\bin folder. Moreover, the runner uses the following command-line arguments:

a: Commands the runner to export all generated test results. However, if you skip this argument, the runner will export only error messages.

A: Commands the runner to organize files with test results into directories and subdirectories that match the test suites and TestCases the runner executes. However, if you skip the -A argument, the runner will save all result files to the directory that the -f argument specifies.

c: Specifies the TestCase to run. Use this argument to narrow down the tests to be executed. However, if you do not specify this argument, the runner will launch all the tests that relate to the parent test suite.

D: Specifies the value of a system property for the test run. The specified value will override the variable’s value during the run. Usage: -D=. However, if the value includes spaces, enclose the entire argument in quotes. To override several variable values, specify the -D argument several times.

d: Specifies the domain that the simulated requests will use for authorization. Certainly, this argument overrides the authorization domain specified in your test properties.

e: Specifies the endpoint to use in test requests. Moreover, the specified endpoint should include the protocol part (for example, https://). This argument overrides the endpoints specified for TestSteps in your test project.

E: Specifies the environment to use in the test run (SoapUI Pro only).

F: Specifies the format of the exported reports. Usage: -F<FormatName>. Supported formats include PDF, XLS, HTML, RTF, CSV, TXT and XML. If the parameter is not specified, PDF is used.

f: Specifies the root directory, where the runner will save test result files. However, if the specified directory does not exist, it will be created. Important: if the directory exists, reports’ files in it will be overwritten.

G: Specifies the value of a global property for the test run. Certainly, the specified value will override the variable’s value during the run. Usage: -G<variable>=<value>. However, if the value includes spaces, enclose the entire argument in quotes.

g: Commands the runner to generate a coverage report (HTML format). Subsequently, to use this argument, you need a SoapUI Pro license. After this, sign up for a free trial to check how it works.

h: Specifies the host and port to use in test requests. Usage: -h<host>:<port>. Moreover, you can specify the host by using its IP address or name. This argument overrides endpoints specified in the project file.

I: Commands the runner to ignore errors. However, if you put this argument to the command line, the test log will contain no information on errors that occur during the test run. In case, if you skip this argument, the runner will stop the run on the first error that occurs and will post full information about the error to the log.

i : Commands to the runner to enable UI-related components. Subsequently, use this command-line argument, when you use the UISupport class in your tests.

J : Commands the runner to create JUnit-style reports with test case properties in them. Meaningful only if the -j argument is also specified.

j : Commands the runner to generate JUnit-compatible reports. Certainly, this argument is similar to the -R”JUnit-style HTML Report” command-line argument.

M : Commands the runner to create an XML file with brief test results. However, this argument does not depend on the other command-line arguments that concern result export: -F, -R and -A.

m : Sets the maximum number of TestStep errors to log for each TestCase.

o : Commands the runner to open the generated reports in your default web browser after the test run is over. Subsequently, to use this functionality, you need a SoapUI Pro license. Sign up for a free trial to check how it works.

P : Specifies the value of a project property for the test run. Certainly, the specified value will override the variable’s value during the run. Usage: -P<variable>=<value>. However, if the value includes spaces, enclose the entire argument in quotes. So, to override several variable values, specify the -P argument several times.

p : Specifies the password to use during the run for authorization. Certainly, this argument overrides the authorization password settings specified in your test project.

R : Specifies the type of the report data. Usage: -R. Report type can be one of the following:

First thing first, TestCase Report – Generates a report in the format that is specified by the -F argument. The runner will save the report files to the directory that the -f argument specifies. Depending on the -A argument value, the files can be organized into subdirectories.
Secondly, JUnit-Style HTML Report – Generates a report as JUnit-style HTML files. When this value is used, the runner ignores the -F and -A arguments.
Subsequently, data Export – Generates XML files with report data. When you use this argument, -F must be XML or must not be specified.

Use the -f argument to specify the directory, where the runner will save generated report files. Note that to use reports in your tests, you need a SoapUI Pro license. If you do not have it, sign up for a free trial to try how it works.

r : Commands the runner to include a summary report into the test log.

S : Commands the runner to save the test project after the test run finishes. This command-line argument may be useful if you store data during the test within the project.

s : Specifies the test suite to run. if you skip this argument, the runner will execute all the test suites in your project.

t : Specifies the workspace settings filed to be used during the test run. If you skip this command-line argument, the runner will use the default file – soapui-settings.xml – that is located in your user directory. In addition, use this argument to specify another settings file for the run. Moreover, it helps you use different proxy, SSL, HTTP and other settings without changing them in your test project.

u : Specifies the user name to be used in test request authorizations. Certainly, this argument overrides user names specified in your test project.

v : Specifies the password for your settings XML file.

w : Specifies the WSS password type. Usage: -w<password type>, where <password type> is of the following:

Firstly, Text – Corresponds to the PasswordText WSS password type.
Subsequently, Digest – Corresponds to the PasswordDigest WSS password type.

x : Specifies the project password, in case the project is encrypted.

Examples

The following command runs all functional tests from the specified project:

testrunner.bat “c:\my projects\my-project.xml”

The following command runs the ‘my test case’ TestCase in your project and creates HTML files in a JUnit-style format:

testrunner.bat -FPDF -R”JUnit-Style HTML Report” -c”my test case” c:\my projects\my-project.xml

Launching the TestRunner from within SoapUI

For getting started with the command line testrunner, SoapUI includes a “Launch TestRunner” action available from Project, TestSuite or TestCase popup menus. This further launches the bundled command-line tools from inside SoapUI. The dialog looks as follows:

Here we’ve specified some initial settings to run the current TestCase, in the Report tab we can specify which reports to generate. Now if we run the testrunner we get:

Scrolling back up in the opened window we can see the actual command issued at the command-line. Further, copy and paste this into your favorite automation tool for rerun these tests as configured.

Exporting parsable results from SoapUI

When integrating SoapUI’s testrunner with Continuous Integration (CI) you can parse the results of an automated test run in one of the following ways:

First of all, you can use the “-r” option of testrunner script to output the results to the console and parse the results from there
Secondly, you can generate a JUnit style report that will create an HTML and XML report files you can parse
Lastly, you can also export your results in CSV or XML format to parse the results in your CI tool

Launching Runners

Although test runners are designed to be run from the command line. Moreover, SoapUI also includes a “Launch TestRunner” action available from the context menu.

The runners are available from the context menu at TestSuite and TestCase levels

Launch TestRunner Tabs Options

Option	Description	Comment
TestSuite	Specifies which TestSuite to run	Default is <all>
TestCase	Specifies which TestCase to run	Default is <all>
Enable UI	Turns on access to the UISupport class
TestRunner Path	Specifies location of TestRunner.bat or TestRunner.sh	Enter manually or use the Browse button
Save Project	Saves project before running
Add Settings	Add the global settings to runner
Project Password	Project password for encrypted projects
user-settings.xml Password	Password for settings file
Ignore Errors	Keep running if errors occur
Save After	Save the project file after tests have been run
Run in-process	Use same process as the GUI