ts
Test Server Framework for testing OTP
This is a framework for testing OTP. The ts
module
implements the interface to all the functionality in the
framework.
The framework is built on top of the Test Server Controller,
test_server_ctrl
, and provides a high level operator
interface. The main features added by the framework are:
- Automatic compilation of test suites and data directories
- Collection of files in central directories and creation of additional HTML pages for better overview.
- Single command interface for running all available tests
- Spawns a new node with correct parameters before starting the test server
- Atomatically creates the parameter file needed when running tests on remote target
More information about the Test Server Framework and how to run test cases can be found in the Test Server User's Guide.
For writing you own test server framework, please turn to the reference manual for the Test Server Controller and chapter named "Writing your own test server framework" in the Test Server User's Guide.
SETUP
To be able to run ts
, you must first `install'
ts
for the current environment. This is done by calling
ts:install/0/1/2
. A file called `variables' is created
and used by ts
when running test suites. It is not
recommended to edit this file, but it is possible to alter if
ts
gets the wrong idea about your environment.
ts:install/0
is used if the target platform is the
same as the controller host, i.e. if you run on "local target"
and no options are needed. Then running ts:install/0
ts
will run an autoconf script for your current
environment and set up the necessary variables needed by the
test suites.
ts:install/1
or ts:install/2
is used if the
target platform is different from the controller host, i.e. if
you run on "remote target" or if special options are required
for your system. VxWorks is currently supported
as remote target platform.
See the reference manual for detailed information about
ts:install/0/1/2
.
Some of the common variables in the 'variables' file are described below. Do not make any assumptions as of what is found in this file, as it may change at any time.
longnames
Set to true if the system is using fully qualified nodenames.platform_id
This is the currently installed platform identification string.platform_filename
This is the name used to create the final save directory for test runs.platform_label
This is the string presented in the generated test results index page.rsh_name
This is the rsh program to use when starting slave or peer nodes on a remote host.erl_flags
Compile time flags used when compiling test suites.erl_release
The Erlang/OTP release being tested.'EMULATOR'
The emulator being tested (e.g. beam)'CPU'
The CPU in the machine running the tests, e.g. sparc.target_host
The target host nameos
The target operating system, e.g. solaris2.8target
The current target platform, e.g. sparc-sun-solaris2.8
RUNNING TESTS
After installing ts
, you can run your test with the
ts:run/0/1/2/3/4
functions. These functions, however,
require a special directory structure to be able to find your
test suites. Both the test server and all tests must be located
under your $TESTROOT directory. The test server implementation
shall be located in the directory $TESTROOT/test_server
and for each application there must be a directory named
$TESTROOT/<application>_test
containing the .spec file
and all test suites and data directories for the
application. Note that there shall only be one .spec file for
each application.
$TESTROOT/test_server
must be the current directory
when calling the ts:run/*
function.
All available tests can be found with ts:tests()
. This
will list all applications for which a test specification file
../<application>_test/<application>.spec
can be found.
To run all these tests, use ts:run()
.
To run one or some of the tests, use ts:run(Tests)
,
where Tests
is the name of the application you want to
test, or a list of such names.
To run one test suite within a test, use
ts:run(Test,Suite)
.
To run one test case within a suite, use
ts:run(Test,Suite,Case)
To all these functions, you can also add a list of
options. Please turn to the reference manual for the ts
module to see the valid options to use.
The function ts:help()
displays some simple help for
the functions in ts
. Use this for quick reference.
LOG FILES
As the execution of the test suites go on, events are logged in four different ways:
- Text to the operator's console.
- Suite related information is sent to the major log file.
- Case related information is sent to the minor log file.
- The HTML log file gets updated with test results.
Typically the operator, who may run hundreds or thousands of test cases, doesn't want to fill the screen with details about/from the specific test cases. By default, the operator will only see:
- A confirmation that the test has started.
- A small note about each failed test case.
- A summary of all the run test cases.
- A confirmation that the test run is complete
- Some special information like error reports and progress
reports, printouts written with erlang:display/1 or io:format/3
specifically addressed to somewhere other than
standard_io
.
This is enough for the operator to know, and if he wants to dig in deeper into a specific test case result, he can do so by following the links in the HTML presentation to take a look in the major or minor log files.
A detailed report of the entire test suite is stored in the major logfile, the exact reason for failure, time spent etc.
The HTML log file is a summary of the major log file, but gives a much better overview of the test run. It also has links to every test case's log file for quick viewing with a HTML browser.
The minor log file contain full details of every single test case, each one in a separate file. This way the files should be easy to compare with previous test runs, even if the set of test cases change.
Which information that goes where is user configurable via the test server controller. Three threshold values determine what comes out on screen, and in the major or minor log files. The contents that goes to the HTML log file is fixed, and cannot be altered.
Functions
install() -> ok | {error, Reason}
install(TargetSystem) -> ok | {error, Reason}
install(Opts) -> ok | {error, Reason}
install(TargetSystem,Opts) -> ok | {error, Reason}
TargetSystem = {Architecture, TargetHost}
Architecture = atom() or string()
TargetHost = atom() or string()
Opts = list()
Installs and configures the Test Server Framework for
running test suites. If a remote host is to be used, the
TargetSystem
argument must be given so that "cross
installation" can be done. This should be used for testing on
VxWorks. Installation is required for any of the
functions in ts
to work.
Opts may be one or more of
{longnames, Bool}
Use fully qualified hostnames for test_server and slave nodes. Bool istrue
orfalse
(default).{verbose, Level}
Verbosity level for test server output, set to 0, 1 or 2, where 0 is quiet(default).{hosts, Hosts}
This is a list of available hosts on which to start slave nodes. It is used when the{remote, true}
option is given to thetest_server:start_node/3
function. Also, if{require_nodenames, Num}
is contained in a test specification file, the generated nodenames will be spread over all hosts given in thisHosts
list. The hostnames are given as atoms or strings.{slavetargets, SlaveTarges}
For VxWorks only. This is a list of available hosts where slave nodes can be started. This is necessary because only one node can run per host in the VxWorks environment. This is not the same as{hosts, Hosts}
because it is used for all slave nodes - not only the ones started with{remote, true}
. The hostnames are given as atoms or strings.{crossroot, TargetErlRoot}
Erlang root directory on target host
This option is mandatory for remote targets{master, {MasterHost, MasterCookie}}
If target is remote and the target node is started as a slave node, this option indicates which master and cookie to use. The given master will also be used as master for slave nodes started withtest_server:start_node/3
. It is expected that theerl_boot_server
is started on the master node before the test is run. If this option is not given, the test server controller node is used as master and theerl_boot_server
is automatically started.{erl_start_args, ArgString}
Additional arguments to be used when starting the test server controller node.ArgString
will be appended to the command line when starting the erlang node. Note that this will only affect the startup of the controller node, i.e. not the target node or any slave nodes startet from a test case.{ipv6_hosts, HostList}
This option will be inserted in theConfig
parameter for each test case.HostList
is a list of hosts supporting IPv6.
help() -> ok
Presents simple help on the functions in ts
. Useful
for quick reference.
tests() -> Tests
Returns the list of available tests. This is actually just a list of all test specification files found by looking up "../*_test/*.spec".
In each ../Name_test/ directory there should be one test specification file named Name.spec.
run() -> ok | {error, Reason}
run([all_tests|Opts])
run(Specs)
run(Specs, Opts)
run(Spec, Module)
run(Spec, Module, Opts)
run(Spec, Module, Case)
run(Spec, Module, Case, Opts)
Specs = Spec | [Spec]
Spec = atom()
Module = atom()
Case = atom()
Opts = [Opt]
Opt = batch | verbose | {verbose, Level} | {vars, Vars} | keep_topcase | cover | cover_details |{cover,CoverFile} | {cover_details,CoverFile} | {trace, TraceSpec}
Level = integer(); 0 means silent
Vars = list() of key-value tuples
CoverFile = string(); name of file listing modules to exclude from or include in cover compilation. The name must include full path to the file.
Reason = term()
This function runs test suite(s)/case(s). To be able to run
any tests, ts:install must first be called to create the
variables
file needed. To run a whole test specification,
only specify the name of the test specification, and all test
suite modules belonging to that test spec will be run. To run
a single module in a test specification, use the Module
argument to specify the name of the module to run and all test
cases in that module will be run, and to run a specified test
case, specify the name of the test case using the Case
argument. If called with no argument, all test specifications
available will be run. Use ts:tests/0 to see the available
test specifications.
If the batch
option is not given, a new xterm is
started (unix) when ts:run
is called.
The verbose
option sets the verbosity level for test
server output. This has the same effect as if given to
ts:install/1/2
The vars
option can be used for adding configuration
variables that are not in the variables
file generated
during installation. Can be any of the Opts
valid for
ts:install/1/2
.
The keep_topcase
option forces ts
to keep the
topcase in your test specification file as is. This option can
only be used if you don't give the Module
or
Case
parameters to ts:run
. The
keep_topcase
option is necessary if your topcase
contains anything other than {dir,"../<Name>_test"}
. If
the option is not used, ts
will modify your topcase.
The cover
and cover_details
options indicates
that the test shall be run with code coverage
analysis. cover_details
means that analysis shall be
done on the most detailed level. If the test is run with a
remote target, this option creates a list of uncovered lines
in each cover compiled module. If the test is run with a local
target, each cover compiled module will be analysed with
cover:analyse_to_file/1
. The cover
options will
only create an overview of all cover compiled modules with the
number of covered and not covered lines.
The CoverFile
which can be given with the
cover
and cover_details
options must be the
filename of a file listing modules to be excluded from or
included in the cover compilation. By default, ts
believes that Spec
is the name of an OTP application
and that all modules in this application shall be cover
compiled. The CoverFile
can exclude modules that belong
to the application and add modules that don't belong to the
application. The file can have the following entries:
{exclude, all | ExcludeModuleList}. {include, IncludeModuleList}.
Note that each line must end with a full
stop. ExcludeModuleList
and IncludeModuleList
are lists of atoms, where each atom is a module name.
If the cover
or cover_details
options are
given on their own, the directory ../<Spec>_test
is
searched for a CoverFile
named <Spec>.cover
. If
this file is not found, Spec
is assumed to be the name
of an OTP application, and all modules in the ebin
directory for the application are cover compiled. The
ebin
directory is found by adding ebin
to
code:lib_dir(Spec)
.
The same cover compiled code will be loaded on all slave or
peer nodes started with test_server:start_node/3
. The
exception is nodes that run an old version of OTP. If the loading
fails, the node will simply not be a part of the coverage
analysis. Note that slave and peer nodes must be stopped with
test_server:stop_node/1
for the node to be part of the
coverage analysis, else the test server will not be able to
fetch coverage data from the node.
The trace
option is used to turn on call trace on
target and on slave or peer nodes started with
test_server:start_node/3
. TraceSpec
can be the
name of a trace information file, or a list of elements like
the ones in a trace information file. Please turn to the
reference manual for test_server_ctrl:trc/1
for details
about the trace information file.
cross_cover_analyse(Level) -> ok
cross_cover_analyse([Level]) -> ok
Analyse cover data collected from all tests.
See test_server_ctrl:cross_cover_analyse/1
r() -> ok
r(Opts) -> ok
r(SpecOrSuite) -> ok
r(SpecOrSuite,Opts) -> ok
r(Suite,Case) -> ok
r(Suite,Case,Opts) -> ok
SpecOrSuite = Spec | Suite
Spec = string()
Suite = atom()
Case = atom()
Opts = [Opt]
Opt = {Cover,AppOrCoverFile} | {Cover,Application,CoverFile}
Cover = cover | cover_details
AppOrCoverFile = Application | CoverFile
Application = atom()
CoverFile = string()
This function can be used to run a test suites or test
cases directly, without any of the additional features added
by the test server framework. It is simply a wrapper function
for the add_dir
, add_spec
, add_module
and
add_case
functions in test_server_ctrl
:
r() -> add_dir(".")
r(Spec) -> add_spec(Spec)
r(Suite) -> add_module(Suite)
r(Suite,Case) -> add_case(Suite,Case)
To use this function, it is required that the test suite is
compiled and in the code path of the node where the function
is called. The function can be used without having ts
installed.
For information about the cover
and
cover_details
options, see test_server_ctrl:cover/2/3
.
index() -> ok | {error, Reason}
Reason = term()
This function updates the local index page. This can be useful if a previous test run was not completed and the index is incomplete.
clean() -> ok
clean(all) -> ok
This function cleans up log directories created when
running test cases. clean/0
cleans up all but the last
run of each application. clean/1
cleans up all test
runs found.
estone() -> ok | {error, Reason}
estone(Opts) -> ok
This function runs the EStone test. It is a shortcut for
running the test suite estone_SUITE
in the
kernel
application.
Opts
is the same as the Opts
argument for the
ts:run
functions.
Makfile.src in Data Directory
If a data directory contains code which must be compiled before
the test suite is run, a makefile source called
Makefile.src
can be placed in the data directory. This file
will be converted to a valid makefile by ts:run/0/1/2/3/4
.
The reason for generating the makefile is that you can use
variables from the variables
file which was generated by
ts:install/0/1/2
. All occurrences of @Key@
in
Makefile.src
is substituted by the Value
from
{Key,Value}
found in the variables
file. Example:
Cut from variables
:
... {'EMULATOR',"beam"}. {'CFLAGS',"-g -O2"}. {'LD',"$(CC) $(CFLAGS)"}. {'CC',"gcc"}. ...
Makefile.src
for compiling erlang code could look
something like this:
EFLAGS=+debug_info all: ordsets1.@EMULATOR@ ordsets1.@EMULATOR@: ordsets1.erl erlc $(EFLAGS) ordsets1.erl
Makefile.src
for compiling c code could look
something like this:
CC = @CC@ LD = @LD@ CFLAGS = @CFLAGS@ -I@erl_include@ @DEFS@ CROSSLDFLAGS = @CROSSLDFLAGS@ PROGS = nfs_check@exe@ all: $(PROGS) nfs_check@exe@: nfs_check@obj@ $(LD) $(CROSSLDFLAGS) -o nfs_check nfs_check@obj@ @LIBS@ nfs_check@obj@: nfs_check.c $(CC) -c -o nfs_check@obj@ $(CFLAGS) nfs_check.c