common_test
A framework for automated testing of arbitrary target nodes
The Common Test framework is an environment for implementing and performing automatic and semi-automatic execution of test cases. Common Test uses the OTP Test Server as engine for test case execution and logging.
In brief, Common Test supports:
- Automated execution of test suites (sets of test cases).
- Logging of the events during execution.
- HTML presentation of test suite results.
- HTML presentation of test suite code.
- Support functions for test suite authors.
- Step by step execution of test cases.
The following sections describe the mandatory and optional test suite functions Common Test will call during test execution. For more details see Common Test User's Guide.
TEST CASE CALLBACK FUNCTIONS
The following functions define the callback interface for a test suite.
Functions
Module:all() -> TestCases | {skip,Reason}
TestCases = [atom() | {group,GroupName}]
Reason = term()
GroupName = atom()
MANDATORY
This function must return the list of all test cases and test
case groups in the test suite module that are to be executed.
This list also specifies the order the cases and groups will
be executed by Common Test. A test case is represented by an atom,
the name of the test case function. A test case group is
represented by a {group,GroupName}
tuple, where
GroupName
, an atom, is the name of the group (defined
with groups/0
).
If {skip,Reason}
is returned, all test cases
in the module will be skipped, and the Reason
will
be printed on the HTML result page.
For details on groups, see Test case groups in the User's Guide.
Module:groups() -> GroupDefs
GroupDefs = [Group]
Group = {GroupName,Properties,GroupsAndTestCases}
GroupName = atom()
Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
TestCase = atom()
Shuffle = shuffle | {shuffle,Seed}
Seed = {integer(),integer(),integer()}
RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail | repeat_until_any_ok | repeat_until_any_fail
N = integer() | forever
OPTIONAL
See Test case groups in the User's Guide for details.
Module:suite() -> [Info]
Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}
Time = TimeVal | TimeFunc
TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()}
TimeFunc = {Mod,Func,Args} | Fun
MilliSec = integer()
Mod = atom()
Func = atom()
Args = list()
Fun = fun()
Required = Key | {Key,SubKeys}
Key = atom()
SubKeys = SubKey | [SubKey]
SubKey = atom()
Name = atom()
UserData = term()
Conns = [atom()]
CSSFile = string()
CTHs = [CTHModule | {CTHModule, CTHInitArgs} | {CTHModule, CTHInitArgs, CTHPriority}]
CTHModule = atom()
CTHInitArgs = term()
OPTIONAL
This is the test suite info function. It is supposed to return a list of tagged tuples that specify various properties regarding the execution of this test suite (common for all test cases in the suite).
The timetrap
tag sets the maximum time each
test case is allowed to take (including init_per_testcase/2
and end_per_testcase/2
). If the timetrap time is
exceeded, the test case fails with reason
timetrap_timeout
. If a TimeFunc
function is specified,
it will be called initially and must return a value on
TimeVal
format.
The require
tag specifies configuration variables
that are required by test cases in the suite. If the required
configuration variables are not found in any of the
configuration files, all test cases are skipped. For more
information about the 'require' functionality, see the
reference manual for the function
ct:require/[1,2]
.
With userdata
, it is possible for the user to
specify arbitrary test suite related information which can be
read by calling ct:userdata/2
.
The ct_hooks
tag specifies which
Common Test Hooks
are to be run together with this suite.
Other tuples than the ones defined will simply be ignored.
For more information about the test suite info function, see Test suite info function in the User's Guide.
Module:init_per_suite(Config) -> NewConfig | {skip,Reason} | {skip_and_save,Reason,SaveConfig}
Config = NewConfig = SaveConfig = [{Key,Value}]
Key = atom()
Value = term()
Reason = term()
OPTIONAL
This function is called as the first function in the
suite. It typically contains initialization which is common for
all test cases in the suite, and which shall only be done
once. The Config
parameter is the configuration
which can be modified here. Whatever is returned from this
function is given as Config
to all configuration functions
and test cases in the suite. If {skip,Reason}
is returned, all test cases in the suite will be skipped and Reason
printed in the overview log for the suite.
For information on save_config
and skip_and_save
,
please see
Dependencies
between Test Cases and Suites in the User's Guide.
Module:end_per_suite(Config) -> void() | {save_config,SaveConfig}
Config = SaveConfig = [{Key,Value}]
Key = atom()
Value = term()
OPTIONAL
This function is called as the last test case in the
suite. It is meant to be used for cleaning up after init_per_suite/1
.
For information on save_config
, please see
Dependencies between
Test Cases and Suites in the User's Guide.
Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason}
GroupName = atom()
Config = NewConfig = [{Key,Value}]
Key = atom()
Value = term()
Reason = term()
OPTIONAL
This function is called before execution of a test case group.
It typically contains initialization which is common for
all test cases in the group, and which shall only be performed
once. GroupName
is the name of the group, as specified in
the group definition (see groups/0
). The Config
parameter is the configuration which can be modified here.
Whatever is returned from this function is given as Config
to all test cases in the group. If {skip,Reason}
is returned,
all test cases in the group will be skipped and Reason
printed
in the overview log for the group.
For information about test case groups, please see Test case groups chapter in the User's Guide.
Module:end_per_group(GroupName, Config) -> void() | {return_group_result,Status}
GroupName = atom()
Config = [{Key,Value}]
Key = atom()
Value = term()
Status = ok | skipped | failed
OPTIONAL
This function is called after the execution of a test case group is finished.
It is meant to be used for cleaning up after init_per_group/2
.
By means of {return_group_result,Status}
, it is possible to return a
status value for a nested sub-group. The status can be retrieved in
end_per_group/2
for the group on the level above. The status will also
be used by Common Test for deciding if execution of a group should proceed in
case the property sequence
or repeat_until_*
is set.
For more information about test case groups, please see Test case groups chapter in the User's Guide.
Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason} | {skip,Reason}
TestCase = atom()
Config = NewConfig = [{Key,Value}]
Key = atom()
Value = term()
Reason = term()
OPTIONAL
This function is called before each test case. The
TestCase
argument is the name of the test case, and
Config
(list of key-value tuples) is the configuration
data that can be modified here. The NewConfig
list returned
from this function is given as Config
to the test case.
If {fail,Reason}
is returned, the test case is
marked as failed without being executed. If {skip,Reason}
is
returned, the test case will be skipped and Reason
printed
in the overview log for the suite.
Module:end_per_testcase(TestCase, Config) -> void() | {fail,Reason} | {save_config,SaveConfig}
TestCase = atom()
Config = SaveConfig = [{Key,Value}]
Key = atom()
Value = term()
Reason = term()
OPTIONAL
This function is called after each test case, and can be used
to clean up after init_per_testcase/2
and the test case.
Any return value (besides {fail,Reason}
and {save_config,SaveConfig}
)
is ignored. By returning {fail,Reason}
, TestCase
will be marked as
failed (even though it was actually successful in the sense that it returned
a value instead of terminating). For information on save_config
, please see
Dependencies between
Test Cases and Suites in the User's Guide
Module:Testcase() -> [Info]
Info = {timetrap,Time} | {require,Required} | {require,Name,Required} | {userdata,UserData} | {silent_connections,Conns}
Time = TimeVal | TimeFunc
TimeVal = MilliSec | {seconds,integer()} | {minutes,integer()} | {hours,integer()}
TimeFunc = {Mod,Func,Args} | Fun
MilliSec = integer()
Mod = atom()
Func = atom()
Args = list()
Fun = fun()
Required = Key | {Key,SubKeys}
Key = atom()
SubKeys = SubKey | [SubKey]
SubKey = atom()
Name = atom()
UserData = term()
Conns = [atom()]
OPTIONAL
This is the test case info function. It is supposed to return a list of tagged tuples that specify various properties regarding the execution of this particular test case.
The timetrap
tag sets the maximum time the
test case is allowed to take. If the timetrap time is
exceeded, the test case fails with reason
timetrap_timeout
. init_per_testcase/2
and end_per_testcase/2
are included in the
timetrap time. If a TimeFunc
function is specified,
it will be called before the test case (or init_per_testcase/2
)
and must return a value on TimeVal
format.
The require
tag specifies configuration variables
that are required by the test case. If the required
configuration variables are not found in any of the
configuration files, the test case is skipped. For more
information about the 'require' functionality, see the
reference manual for the function
ct:require/[1,2]
.
If timetrap
and/or require
is not set, the
default values specified in the suite/0
return list
will be used.
With userdata
, it is possible for the user to
specify arbitrary test case related information which can be
read by calling ct:userdata/3
.
Other tuples than the ones defined will simply be ignored.
For more information about the test case info function, see Test case info function in the User's Guide.
Module:Testcase(Config) -> void() | {skip,Reason} | {comment,Comment} | {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit()
Config = SaveConfig = [{Key,Value}]
Key = atom()
Value = term()
Reason = term()
Comment = string()
MANDATORY
This is the implementation of a test case. Here you must
call the functions you want to test, and do whatever you
need to check the result. If something fails, make sure the
function causes a runtime error, or call ct:fail/[0,1]
(which also causes the test case process to crash).
Elements from the Config
parameter can be read
with the ?config
macro. The config
macro is defined in ct.hrl
You can return {skip,Reason}
if you decide not to
run the test case after all. Reason
will then be
printed in 'Comment' field on the HTML result page.
You can return {comment,Comment}
if you wish to
print some information in the 'Comment' field on the HTML
result page.
If the function returns anything else, the test case is considered successful. (The return value always gets printed in the test case log file).
For more information about test case implementation, please see Test cases in the User's Guide.
For information on save_config
and skip_and_save
, please see
Dependencies between
Test Cases and Suites in the User's Guide.