ct
Main user interface for the Common Test framework.
Main user interface for the Common Test framework.
This module implements the command line interface for running tests and some basic functions for common test case issues such as configuration and logging.
Test Suite Support Macros
The config
macro is defined in ct.hrl
. This
macro should be used to retrieve information from the
Config
variable sent to all test cases. It is used with two
arguments, where the first is the name of the configuration
variable you wish to retrieve, and the second is the Config
variable supplied to the test case.
Possible configuration variables include:
data_dir
- Data file directory.priv_dir
- Scratch file directory.Whatever added by
init_per_suite/1
orinit_per_testcase/2
in the test suite.
Functions
abort_current_testcase(Reason) -> ok | {error, ErrorReason}
Reason = term()
ErrorReason = no_testcase_running | parallel_group
When calling this function, the currently executing test case will be aborted. It is the user's responsibility to know for sure which test case is currently executing. The function is therefore only safe to call from a function which has been called (or synchronously invoked) by the test case.
Reason
, the reason for aborting the test case, is printed
in the test case log.
add_config(Callback, Config) -> ok | {error, Reason}
Callback = atom()
Config = string()
Reason = term()
This function loads configuration variables using the
given callback module and configuration string. Callback module
should be either loaded or present in the code part. Loaded
configuration variables can later be removed using
remove_config/2
function.
break(Comment) -> ok | {error, Reason}
Comment = string()
Reason = {multiple_cases_running, TestCases} | 'enable break with release_shell option'
TestCases = [atom()]
This function will cancel any active timetrap and pause the
execution of the current test case until the user calls the
continue/0
function. It gives the user the opportunity
to interact with the erlang node running the tests, e.g. for
debugging purposes or for manually executing a part of the
test case. If a parallel group is executing, break/2
should be called instead.
A cancelled timetrap will not be automatically
reactivated after the break, but must be started exlicitly with
ct:timetrap/1
In order for the break/continue functionality to work,
Common Test must release the shell process controlling stdin.
This is done by setting the release_shell
start option
to true
. See the User's Guide for more information.
break(TestCase, Comment) -> ok | {error, Reason}
TestCase = atom()
Comment = string()
Reason = 'test case not running' | 'enable break with release_shell option'
This function works the same way as break/1
,
only the TestCase
argument makes it possible to
pause a test case executing in a parallel group. The
continue/1
function should be used to resume
execution of TestCase
.
See break/1
for more details.
capture_get(ExclCategories) -> ListOfStrings
ExclCategories = [atom()]
ListOfStrings = [string()]
Return and purge the list of text strings buffered
during the latest session of capturing printouts to stdout.
With ExclCategories
it's possible to specify
log categories that should be ignored in ListOfStrings
.
If ExclCategories = []
, no filtering takes place.
See also: capture_start/0, capture_stop/0, log/3.
capture_start() -> ok
Start capturing all text strings printed to stdout during execution of the test case.
See also: capture_get/1, capture_stop/0.
capture_stop() -> ok
Stop capturing text strings (a session started with
capture_start/0
).
See also: capture_get/1, capture_start/0.
comment(Comment) -> void()
Comment = term()
Print the given Comment
in the comment field in
the table on the test suite result page.
If called several times, only the last comment is printed.
The test case return value {comment,Comment}
overwrites the string set by this function.
comment(Format, Args) -> void()
Format = string()
Args = list()
Print the formatted string in the comment field in the table on the test suite result page.
The Format
and Args
arguments are
used in call to io_lib:format/2
in order to create
the comment string. The behaviour of comment/2
is
otherwise the same as the comment/1
function (see
above for details).
continue() -> ok
This function must be called in order to continue after a
test case (not executing in a parallel group) has called
break/1
.
continue(TestCase) -> ok
TestCase = atom()
This function must be called in order to continue after a
test case has called break/2
. If the paused test case,
TestCase
, executes in a parallel group, this
function - rather than continue/0
- must be used
in order to let the test case proceed.
decrypt_config_file(EncryptFileName, TargetFileName) -> ok | {error, Reason}
EncryptFileName = string()
TargetFileName = string()
Reason = term()
This function decrypts EncryptFileName
, previously
generated with encrypt_config_file/2/3
. The original
file contents is saved in the target file. The encryption key, a
string, must be available in a text file named
.ct_config.crypt
in the current directory, or the
home directory of the user (it is searched for in that order).
decrypt_config_file(EncryptFileName, TargetFileName, KeyOrFile) -> ok | {error, Reason}
EncryptFileName = string()
TargetFileName = string()
KeyOrFile = {key, string()} | {file, string()}
Reason = term()
This function decrypts EncryptFileName
, previously
generated with encrypt_config_file/2/3
. The original
file contents is saved in the target file. The key must have the
the same value as that used for encryption.
encrypt_config_file(SrcFileName, EncryptFileName) -> ok | {error, Reason}
SrcFileName = string()
EncryptFileName = string()
Reason = term()
This function encrypts the source config file with DES3 and
saves the result in file EncryptFileName
. The key,
a string, must be available in a text file named
.ct_config.crypt
in the current directory, or the
home directory of the user (it is searched for in that order).
See the Common Test User's Guide for information about using encrypted config files when running tests.
See the crypto
application for details on DES3
encryption/decryption.
encrypt_config_file(SrcFileName, EncryptFileName, KeyOrFile) -> ok | {error, Reason}
SrcFileName = string()
EncryptFileName = string()
KeyOrFile = {key, string()} | {file, string()}
Reason = term()
This function encrypts the source config file with DES3 and
saves the result in the target file EncryptFileName
.
The encryption key to use is either the value in
{key,Key}
or the value stored in the file specified
by {file,File}
.
See the Common Test User's Guide for information about using encrypted config files when running tests.
See the crypto
application for details on DES3
encryption/decryption.
fail(Reason) -> void()
Reason = term()
Terminate a test case with the given error
Reason
.
fail(Format, Args) -> void()
Format = string()
Args = list()
Terminate a test case with an error message specified
by a format string and a list of values (used as arguments to
io_lib:format/2
).
get_config(Required) -> Value
Equivalent to get_config(Required, undefined, []).
get_config(Required, Default) -> Value
Equivalent to get_config(Required, Default, []).
get_config(Required, Default, Opts) -> ValueOrElement
Required = KeyOrName | {KeyOrName, SubKey} | {KeyOrName, SubKey, SubKey}
KeyOrName = atom()
SubKey = atom()
Default = term()
Opts = [Opt] | []
Opt = element | all
ValueOrElement = term() | Default
Read config data values.
This function returns the matching value(s) or config element(s),
given a config variable key or its associated name
(if one has been specified with require/2
or a
require statement).
Example, given the following config file:
{unix,[{telnet,IpAddr}, {user,[{username,Username}, {password,Password}]}]}.
ct:get_config(unix,Default) ->
[{telnet,IpAddr},
{user, [{username,Username},
{password,Password}]}]
ct:get_config({unix,telnet},Default) -> IpAddr
ct:get_config({unix,user,username},Default) -> Username
ct:get_config({unix,ftp},Default) -> Default
ct:get_config(unknownkey,Default) -> Default
If a config variable key has been associated with a name (by
means of require/2
or a require statement), the name
may be used instead of the key to read the value:
ct:require(myuser,{unix,user}) -> ok.
ct:get_config(myuser,Default) ->
[{username,Username},
{password,Password}]
If a config variable is defined in multiple files and you want to
access all possible values, use the all
option. The
values will be returned in a list and the order of the elements
corresponds to the order that the config files were specified at
startup.
If you want config elements (key-value tuples) returned as result
instead of values, use the element
option.
The returned elements will then be on the form {Required,Value}
See also: get_config/1, get_config/2, require/1, require/2.
get_status() -> TestStatus | {error, Reason} | no_tests_running
TestStatus = [StatusElem]
StatusElem = {current, TestCaseInfo} | {successful, Successful} | {failed, Failed} | {skipped, Skipped} | {total, Total}
TestCaseInfo = {Suite, TestCase} | [{Suite, TestCase}]
Suite = atom()
TestCase = atom()
Successful = integer()
Failed = integer()
Skipped = {UserSkipped, AutoSkipped}
UserSkipped = integer()
AutoSkipped = integer()
Total = integer()
Reason = term()
Returns status of ongoing test. The returned list contains info about which test case is currently executing (a list of cases when a parallel test case group is executing), as well as counters for successful, failed, skipped, and total test cases so far.
get_target_name(Handle) -> {ok, TargetName} | {error, Reason}
Handle = handle()
TargetName = target_name()
Return the name of the target that the given connection belongs to.
get_timetrap_info() -> {Time, Scale}
Time = integer() | infinity
Scale = true | false
Read info about the timetrap set for the current test case.
Scale
indicates if Common Test will attempt to automatically
compensate timetraps for runtime delays introduced by e.g. tools like
cover.
install(Opts) -> ok | {error, Reason}
Opts = [Opt]
Opt = {config, ConfigFiles} | {event_handler, Modules} | {decrypt, KeyOrFile}
ConfigFiles = [ConfigFile]
ConfigFile = string()
Modules = [atom()]
KeyOrFile = {key, Key} | {file, KeyFile}
Key = string()
KeyFile = string()
Install config files and event handlers.
Run this function once before first test.
Example:
install([{config,["config_node.ctc","config_user.ctc"]}])
.
Note that this function is automatically run by the
ct_run
program.
listenv(Telnet) -> [Env]
Telnet = term()
Env = {Key, Value}
Key = string()
Value = string()
Performs the listenv command on the given telnet connection and returns the result as a list of Key-Value pairs.
log(Format) -> ok
Equivalent to log(default, 50, Format, []).
log(X1, X2) -> ok
X1 = Category | Importance | Format
X2 = Format | Args
Equivalent to log(Category, Importance, Format, Args).
log(X1, X2, X3) -> ok
X1 = Category | Importance
X2 = Importance | Format
X3 = Format | Args
Equivalent to log(Category, Importance, Format, Args).
log(Category, Importance, Format, Args) -> ok
Category = atom()
Importance = integer()
Format = string()
Args = list()
Printout from a test case to the log file.
This function is meant for printing a string directly from a test case to the test case log file.
Default Category
is default
,
default Importance
is ?STD_IMPORTANCE
,
and default value for Args
is []
.
Please see the User's Guide for details on Category
and Importance
.
make_priv_dir() -> ok | {error, Reason}
Reason = term()
If the test has been started with the create_priv_dir option set to manual_per_tc, in order for the test case to use the private directory, it must first create it by calling this function.
notify(Name, Data) -> ok
Name = atom()
Data = term()
Sends a asynchronous notification of type Name
with
Data
to the common_test event manager. This can later be
caught by any installed event manager.
See also: gen_event(3).
pal(Format) -> ok
Equivalent to pal(default, 50, Format, []).
pal(X1, X2) -> ok
X1 = Category | Importance | Format
X2 = Format | Args
Equivalent to pal(Category, Importance, Format, Args).
pal(X1, X2, X3) -> ok
X1 = Category | Importance
X2 = Importance | Format
X3 = Format | Args
Equivalent to pal(Category, Importance, Format, Args).
pal(Category, Importance, Format, Args) -> ok
Category = atom()
Importance = integer()
Format = string()
Args = list()
Print and log from a test case.
This function is meant for printing a string from a test case, both to the test case log file and to the console.
Default Category
is default
,
default Importance
is ?STD_IMPORTANCE
,
and default value for Args
is []
.
Please see the User's Guide for details on Category
and Importance
.
parse_table(Data) -> {Heading, Table}
Data = [string()]
Heading = tuple()
Table = [tuple()]
Parse the printout from an SQL table and return a list of tuples.
The printout to parse would typically be the result of a
select
command in SQL. The returned
Table
is a list of tuples, where each tuple is a row
in the table.
Heading
is a tuple of strings representing the
headings of each column in the table.
print(Format) -> ok
Equivalent to print(default, 50, Format, []).
print(X1, X2) -> ok
X1 = Category | Importance | Format
X2 = Format | Args
Equivalent to print(Category, Importance, Format, Args).
print(X1, X2, X3) -> ok
X1 = Category | Importance
X2 = Importance | Format
X3 = Format | Args
Equivalent to print(Category, Importance, Format, Args).
print(Category, Importance, Format, Args) -> ok
Category = atom()
Importance = integer()
Format = string()
Args = list()
Printout from a test case to the console.
This function is meant for printing a string from a test case to the console.
Default Category
is default
,
default Importance
is ?STD_IMPORTANCE
,
and default value for Args
is []
.
Please see the User's Guide for details on Category
and Importance
.
reload_config(Required) -> ValueOrElement
Required = KeyOrName | {KeyOrName, SubKey} | {KeyOrName, SubKey, SubKey}
KeyOrName = atom()
SubKey = atom()
ValueOrElement = term()
Reload config file which contains specified configuration key.
This function performs updating of the configuration data from which the given configuration variable was read, and returns the (possibly) new value of this variable.
Note that if some variables were present in the configuration but are not loaded using this function, they will be removed from the configuration table together with their aliases.
remove_config(Callback, Config) -> ok
Callback = atom()
Config = string()
Reason = term()
This function removes configuration variables (together with their aliases) which were loaded with specified callback module and configuration string.
require(Required) -> ok | {error, Reason}
Required = Key | {Key, SubKeys} | {Key, SubKey, SubKeys}
Key = atom()
SubKeys = SubKey | [SubKey]
SubKey = atom()
Check if the required configuration is available. It is possible
to specify arbitrarily deep tuples as Required
. Note that it is
only the last element of the tuple which can be a list of SubKey
s.
Example 1: require the variable myvar
:
ok = ct:require(myvar).
In this case the config file must at least contain:
{myvar,Value}.
Example 2: require the key myvar
with
subkeys sub1
and sub2
:
ok = ct:require({myvar,[sub1,sub2]}).
In this case the config file must at least contain:
{myvar,[{sub1,Value},{sub2,Value}]}.
Example 3: require the key myvar
with
subkey sub1
with subsub1
:
ok = ct:require({myvar,sub1,sub2}).
In this case the config file must at least contain:
{myvar,[{sub1,[{sub2,Value}]}]}.
See also: get_config/1, get_config/2, get_config/3, require/2.
require(Name, Required) -> ok | {error, Reason}
Name = atom()
Required = Key | {Key, SubKey} | {Key, SubKey, SubKey}
SubKey = Key
Key = atom()
Check if the required configuration is available, and give it
a name. The semantics for Required
is the same as in
required/1
except that it is not possible to specify a list
of SubKey
s.
If the requested data is available, the sub entry will be
associated with Name
so that the value of the element
can be read with get_config/1,2
provided
Name
instead of the whole Required
term.
Example: Require one node with a telnet connection and an
ftp connection. Name the node a
:
ok = ct:require(a,{machine,node}).
All references to this node may then use the node name. E.g. you can fetch a file over ftp like this:
ok = ct:ftp_get(a,RemoteFile,LocalFile).
For this to work, the config file must at least contain:
{machine,[{node,[{telnet,IpAddr},{ftp,IpAddr}]}]}.
Note!
The behaviour of this function changed radically in common_test
1.6.2. In order too keep some backwards compatability it is still possible
to do:
ct:require(a,{node,[telnet,ftp]}).
This will associate the name a
with the top level node
entry.
For this to work, the config file must at least contain:
{node,[{telnet,IpAddr},{ftp,IpAddr}]}.
See also: get_config/1, get_config/2, get_config/3, require/1.
run(TestDirs) -> Result
TestDirs = TestDir | [TestDir]
Run all test cases in all suites in the given directories.
See also: run/3.
run(TestDir, Suite) -> Result
Run all test cases in the given suite.
See also: run/3.
run(TestDir, Suite, Cases) -> Result
TestDir = string()
Suite = atom()
Cases = atom() | [atom()]
Result = [TestResult] | {error, Reason}
Run the given test case(s).
Requires that ct:install/1
has been run first.
Suites (*_SUITE.erl) files must be stored in
TestDir
or TestDir/test
. All suites
will be compiled when test is run.
run_test(Opts) -> Result
Opts = [OptTuples]
OptTuples = {dir, TestDirs} | {suite, Suites} | {group, Groups} | {testcase, Cases} | {spec, TestSpecs} | {join_specs, Bool} | {label, Label} | {config, CfgFiles} | {userconfig, UserConfig} | {allow_user_terms, Bool} | {logdir, LogDir} | {silent_connections, Conns} | {stylesheet, CSSFile} | {cover, CoverSpecFile} | {cover_stop, Bool} | {step, StepOpts} | {event_handler, EventHandlers} | {include, InclDirs} | {auto_compile, Bool} | {create_priv_dir, CreatePrivDir} | {multiply_timetraps, M} | {scale_timetraps, Bool} | {repeat, N} | {duration, DurTime} | {until, StopTime} | {force_stop, ForceStop} | {decrypt, DecryptKeyOrFile} | {refresh_logs, LogDir} | {logopts, LogOpts} | {verbosity, VLevels} | {basic_html, Bool} | {ct_hooks, CTHs} | {enable_builtin_hooks, Bool} | {release_shell, Bool}
TestDirs = [string()] | string()
Suites = [string()] | [atom()] | string() | atom()
Cases = [atom()] | atom()
Groups = GroupNameOrPath | [GroupNameOrPath]
GroupNameOrPath = [atom()] | atom() | all
TestSpecs = [string()] | string()
Label = string() | atom()
CfgFiles = [string()] | string()
UserConfig = [{CallbackMod, CfgStrings}] | {CallbackMod, CfgStrings}
CallbackMod = atom()
CfgStrings = [string()] | string()
LogDir = string()
Conns = all | [atom()]
CSSFile = string()
CoverSpecFile = string()
StepOpts = [StepOpt] | []
StepOpt = config | keep_inactive
EventHandlers = EH | [EH]
EH = atom() | {atom(), InitArgs} | {[atom()], InitArgs}
InitArgs = [term()]
InclDirs = [string()] | string()
CreatePrivDir = auto_per_run | auto_per_tc | manual_per_tc
M = integer()
N = integer()
DurTime = string(HHMMSS)
StopTime = string(YYMoMoDDHHMMSS) | string(HHMMSS)
ForceStop = skip_rest | Bool
DecryptKeyOrFile = {key, DecryptKey} | {file, DecryptFile}
DecryptKey = string()
DecryptFile = string()
LogOpts = [LogOpt]
LogOpt = no_nl | no_src
VLevels = VLevel | [{Category, VLevel}]
VLevel = integer()
Category = atom()
CTHs = [CTHModule | {CTHModule, CTHInitArgs}]
CTHModule = atom()
CTHInitArgs = term()
Result = {Ok, Failed, {UserSkipped, AutoSkipped}} | TestRunnerPid | {error, Reason}
Ok = integer()
Failed = integer()
UserSkipped = integer()
AutoSkipped = integer()
TestRunnerPid = pid()
Reason = term()
Run tests as specified by the combination of options in Opts
.
The options are the same as those used with the
ct_run
program.
Note that here a TestDir
can be used to point out the path to
a Suite
. Note also that the option testcase
corresponds to the -case
option in the ct_run
program. Configuration files specified in Opts
will be
installed automatically at startup.
TestRunnerPid
is returned if release_shell == true
(see break/1
for details).
Reason
indicates what type of error has been encountered.
run_testspec(TestSpec) -> Result
TestSpec = [term()]
Result = {Ok, Failed, {UserSkipped, AutoSkipped}} | {error, Reason}
Ok = integer()
Failed = integer()
UserSkipped = integer()
AutoSkipped = integer()
Reason = term()
Run test specified by TestSpec
. The terms are
the same as those used in test specification files.
Reason
indicates what type of error has been encountered.
sleep(Time) -> ok
Time = {hours, Hours} | {minutes, Mins} | {seconds, Secs} | Millisecs | infinity
Hours = integer()
Mins = integer()
Secs = integer()
Millisecs = integer() | float()
This function, similar to timer:sleep/1
, suspends the test
case for specified time. However, this function also multiplies
Time
with the 'multiply_timetraps' value (if set) and under
certain circumstances also scales up the time automatically
if 'scale_timetraps' is set to true (default is false).
start_interactive() -> ok
Start CT in interactive mode.
From this mode all test case support functions can be executed
directly from the erlang shell. The interactive mode can also be
started from the OS command line with ct_run -shell
[-config File...]
.
If any functions using "required config data" (e.g. telnet or
ftp functions) are to be called from the erlang shell, config data
must first be required with ct:require/2
.
Example:
> ct:require(unix_telnet, unix).
ok
> ct_telnet:open(unix_telnet).
{ok,<0.105.0>}
> ct_telnet:cmd(unix_telnet, "ls .").
{ok,["ls","file1 ...",...]}
step(TestDir, Suite, Case) -> Result
Case = atom()
Step through a test case with the debugger.
See also: run/3.
step(TestDir, Suite, Case, Opts) -> Result
Case = atom()
Opts = [Opt] | []
Opt = config | keep_inactive
Step through a test case with the debugger. If the
config
option has been given, breakpoints will
be set also on the configuration functions in Suite
.
See also: run/3.
stop_interactive() -> ok
Exit the interactive mode.
See also: start_interactive/0.
sync_notify(Name, Data) -> ok
Name = atom()
Data = term()
Sends a synchronous notification of type Name
with
Data
to the common_test event manager. This can later be
caught by any installed event manager.
See also: gen_event(3).
testcases(TestDir, Suite) -> Testcases | {error, Reason}
TestDir = string()
Suite = atom()
Testcases = list()
Reason = term()
Returns all test cases in the specified suite.
timetrap(Time) -> ok
Time = {hours, Hours} | {minutes, Mins} | {seconds, Secs} | Millisecs | infinity | Func
Hours = integer()
Mins = integer()
Secs = integer()
Millisecs = integer() | float()
Func = {M, F, A} | function()
M = atom()
F = atom()
A = list()
Use this function to set a new timetrap for the running test case.
If the argument is Func
, the timetrap will be triggered
when this function returns. Func
may also return a new
Time
value, which in that case will be the value for the
new timetrap.
userdata(TestDir, Suite) -> SuiteUserData | {error, Reason}
TestDir = string()
Suite = atom()
SuiteUserData = [term()]
Reason = term()
Returns any data specified with the tag userdata
in the list of tuples returned from Suite:suite/0
.
userdata(TestDir, Suite, Case::GroupOrCase) -> TCUserData | {error, Reason}
TestDir = string()
Suite = atom()
GroupOrCase = {group, GroupName} | atom()
GroupName = atom()
TCUserData = [term()]
Reason = term()
Returns any data specified with the tag userdata
in the list of tuples returned from Suite:group(GroupName)
or Suite:Case()
.