Automated Testing
Automated regression testing is provided using test-suite templates to describe actions and expected responses and results. The test-suite command is used to access this feature.
Tests rely on named sessions in order to send commands from different sessions. Each test step can be directed at a different session. The “locking” example in a later section shows how test step commands can be either expected to succeed (e.g., first session attempts <lock> command) and fail (e.g., second session attempts <lock> but the datastore is already locked).
A test suite that only needs to send commands to the current session does not need to use named sessions. The session connection and shutdown steps do not need to be done in the test-suite.
The start-session command is usually used within the 'setup' section for each session required in all of the tests within the test-suite. The server must already be running.
The 'setup' and 'cleanup' sections defined within a test suite are called 'CLI text blocks'. They use a special configuration file syntax which allows maximum flexibility for the test designer. These sections begin and and with square brackets ('[' and ']') instead of angle brackets ('{' and '}'). There are no leafs within these containers. Instead they contain plain command lines exactly like a script.
Test Templates
The test suite file structure is shown below:
test-suite-file: collection of test-suites in a single file. Multiple files can be loaded. If the --autotest CLI parameter is 'true' (default) then yangcli-pro will look for the default test file, called
$HOME/.yumapro/yangcli_pro_tests.conf.test-suite: collection of tests. Each test-suite has an optional 'setup' section which is executed once before the tests are run, and an optional 'cleanup' section that is run after all the tests have run
test: collection of steps to perform a specific test
step: a command to be executed locally or remotely to a specified session. Each step for a remote command can be configured to check the server response received.
YANG File Defining Test Templates
module yumaworks-test {
namespace "http://yumaworks.com/ns/yumaworks-test";
prefix "ywtest";
import yuma-types { prefix yt; }
import yumaworks-extensions { prefix ywx; }
organization "YumaWorks, Inc.";
contact
"Support <[email protected]>";
description
"This module contains data structures representing
replayable test suites for specific use cases and YANG modules
+---------------------+
+-+------------------+ |
| | |
| Test Suite | |
| |--+
+--------------------+
| |
| |
| V
| +---------------------+
| +-+------------------+ |
| | | |
| | Test | |
| | |--+
| +--------------------+
| |
| |
| V
| +---------------------+
| +-+------------------+ |
| | | |
| | Step | |
| | |--+
| +--------------------+
|
V
+---------------------+
+-+------------------+ |
| | |
| Servers | |
| |--+
+--------------------+
|
|
V
+---------------------+
+-+------------------+ |
| | |
| Sessions | |
| |--+
+--------------------+
Copyright (c) 2013 - 2023 YumaWorks, Inc. All rights reserved.
Redistribution and use in source and binary forms, with or
without modification, is permitted pursuant to, and subject
to the license terms contained in, the BSD 3-Clause License
http://opensource.org/licenses/BSD-3-Clause";
revision 2023-03-16 {
description
"Add message-indent parameter";
}
revision 2013-03-31 {
description
"Rename to yumaworks-test";
}
revision 2013-01-21 {
description
"Initial version";
}
typedef response-type {
type enumeration {
enum none {
description "Local command, no <rpc-reply> expected.";
}
enum ok {
description "Expecting the <ok> reply.";
}
enum data {
description "Expecting a data reply.";
}
enum error {
description "Expecting an rpc-error reply.";
}
}
description
"The type of response expected from the server
for this request.";
}
container test-suites {
list test-suite {
key name;
leaf name {
type yt:NcxName;
description
"The test suite name.";
}
leaf description {
type string;
description "Description of this test suite.";
}
container setup {
ywx:cli-text-block;
description "Test suite setup commands";
}
container cleanup {
ywx:cli-text-block;
description "Test suite cleanup commands";
}
leaf-list run-test {
ordered-by user;
type yt:NcxName;
min-elements 1;
description
"The ordered list of test names to run in this test suite.
At least 1 test must be specified.";
}
leaf message-indent {
type int8 {
range "-1 .. 9";
}
default 2;
description "message-indent used for the test.";
}
list test {
description
"The unit-tests that are configured to be run.
At least 1 test must be configured.";
ordered-by user;
min-elements 1;
key name;
leaf name {
type yt:NcxName;
description "The name of this unit test.";
}
leaf description {
type string;
description "Description of this unit test.";
}
leaf-list must-pass {
type yt:NcxName;
description
"The names of the tests that have already been
run and pased for this test to be attempted,
The test will be skipped if any test in the
must-pass list has been attempted and the
test failed.
If the named test has not been run yet
this test will fail and be skipped. If the named
test was skipped, then it will not cause this test
to be skipped, only if it did not run at all or
if it ran and passed.";
}
list step {
description
"A list of test steps to be done in order.";
ordered-by user;
key name;
leaf name {
type string {
length "1 .. 64";
}
description "The name of this unit test step.";
}
leaf description {
type string;
description "Description of this unit test step.";
}
leaf session-name {
type yt:NcxName;
description
"The name of the session to use.
Empty if the test session should be used";
}
leaf result-type {
type response-type;
description
"The expected response type. If this leaf is
missing then any response type is acceptable.";
}
leaf result-error-tag {
when "../result-type = 'error'";
type string;
description
"The error-tag value expected if the result-type
is 'error'. If not set, then any error-tag
value is acceptable.";
}
leaf result-error-apptag {
when "../result-type = 'error'";
type string;
description
"The error-app-tag value expected if the result-type
is 'error'. If not set, then any error-app-tag
value is acceptable.";
}
leaf-list result-error-info {
when "../result-type = 'error'";
type yt:NcxName;
description
"The error-info element name expected if the result-type
is 'error'.";
}
leaf command {
type string;
mandatory true;
description "The yangcli command line string to use";
}
anyxml rpc-reply-data {
when "../result-type = 'data'";
description
"The contents of <rpc-reply> element that are
expected to be returned in the reply.
This element itself represents <rpc-reply>
and any child nodes are the nodes returned
by the server.";
}
} // list step
} // list test
} // list test-suite
} // container test-suites
}
RPC Reply Testing
Each step has a “response-type” field that specifies what kind of RPC reply testing is expected.
none: Do not perform any RPC reply testing, or this step does not send an RPC request to a server
ok: An RPC reply is expected with an <ok> response
data: An RPC reply is expected with some sort of data response. The expected response is stored in an XML file and compared to the actual RPC reply when the step is executed.
Initial files are created automatically during the “record-test” procedure.
The files are stored in the
$HOME/.yumapro/recordtestdirectory.Each filename has the form <suite-name>_<test-name>_<step-number>_record.xml
During the test a temporary file will be created for each data file to save the RPC reply. These files will be deleted when the test suite is completed.
Each file contains a single <rpc-reply> element.
These files can be edited, created, by hand, or even deleted.
Wildcard fields can be used in this file to allow any value to match, removing this node from the comparison test.
error: An RPC reply is expected with an <rpc-error> response
error-tag: The expected error-tag in the <rpc-error> response
error-app-tag: The expected error-app-tag in the <rpc-error> response
result-error-info: A list of names of elements expected in the <error-info> container within the <rpc-error> response
Using Wildcards in Data Response Testing
It is possible simply test for a node within the RPC reply instead of comparing its actual value. Some data such as timestamps and counters may change on every retrieval. Such a node can cause the entire step to fail when comparing the <rpc-reply> contents.
The wildcard string is a sequence of 5 characters (underscore, asterisk, asterisk, asterisk, underscore)
_***_
If this string is used as the value for an element, then that node will be skipped in the comparison.
Example Step:
Suite: s1
Test: t1
Step: 2
Command: sget /netconf-state/datastores
File: $HOME/.yumapro/recordtest/s1_t1_2_record.xml
Original Data File Created During record-test:
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<data>
<netconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
<datastores>
<datastore>
<name>candidate</name>
<last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">2019-12-05T23:21:27Z</last-modified>
</datastore>
<datastore>
<name>running</name>
<last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">2019-12-05T23:21:27Z</last-modified>
</datastore>
</datastores>
</netconf-state>
</data>
</rpc-reply>
Altered Data File Wild Wildcard Strings:
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<data>
<netconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
<datastores>
<datastore>
<name>candidate</name>
<last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">_***_</last-modified>
</datastore>
<datastore>
<name>running</name>
<last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">_***_</last-modified>
</datastore>
</datastores>
</netconf-state>
</data>
</rpc-reply>
Example Test File
test-suites {
test-suite first-test {
description "A set of tests to validate NETCONF locking is working
correctly."
setup [
# example variables
$$server ="test1"
$$admin = "andy"
$$user1 = "andy"
$$pass1 = "fredlow"
$$user2 = "andy2"
$$pass2 = "fredlow"
$$testlog = "$HOME/tests/first-test.log"
$$testmod = "test"
# 2 sessions used in this test; server must already be running
start-session session-A
start-session session-B
]
cleanup [
stop-session session-A
stop-session session-B
]
run-test locks
test locks {
description "Use 2 sessions to test global locks on the
candidate and running datastores. Expects the server
to already be started with --module=test
and --access-control=off."
step 1 {
description "session A locks the running config;
needs to lock candidate too, but does not!"
session-name session-A
result-type ok
command "lock target=running"
}
step 2 {
description "session B tries to lock the running config"
session-name session-B
result-type error
result-error-tag lock-denied
result-error-info session-id
command "lock target=running"
}
step 3 {
description "session A tries to write to the target config"
session-name session-A
result-type ok
command "merge /uint8.1 value=10"
}
step 4 {
description "session B tries to write to the target config
candidate is not locked so this should work"
session-name session-B
result-type ok
command "merge /uint8.1 value=12"
}
step 5 {
description "session B tries to commit the candidate to running
running is locked so this should fail"
session-name session-B
result-type error
result-error-tag in-use
command "commit"
}
step 6 {
description "session A tries to lock the candidate config
this should fail since the candidate is dirty"
session-name session-A
result-type error
result-error-tag resource-denied
command "lock target=candidate"
}
step 7 {
description "session B tries to lock the candidate config
this should fail since the candidate is dirty"
session-name session-B
result-type error
result-error-tag resource-denied
command "lock target=candidate"
}
step 8 {
description "session B issues a discard-changes"
session-name session-B
result-type ok
command "discard-changes"
}
step 9 {
description "session A issues a discard-changes"
session-name session-A
result-type ok
command "discard-changes"
}
step 10 {
description "session B locks the candidate config"
session-name session-B
result-type ok
command "lock target=candidate"
}
step 11 {
description "session A tries to lock the candidate config,
which should fail because it is already locked"
session-name session-A
result-type error
result-error-tag lock-denied
result-error-info session-id
command "lock target=candidate"
}
step 12 {
description "session A tries to write to the target config,
which could fail because candidate is locked"
session-name session-A
result-type error
result-error-tag in-use
command "merge /uint8.1 value=10"
}
step 13 {
description "session B tries to write to the target config"
session-name session-B
result-type ok
command "merge /uint8.1 value=12"
}
step 14 {
description "session A tries to commit the candidate to running,
candidate is locked by B so this should fail"
session-name session-A
result-type error
result-error-tag in-use
command "commit"
}
step 15 {
description "session B tries to commit the candidate to running,
running is locked by A so this should fail"
session-name session-B
result-type error
result-error-tag in-use
command "commit"
}
step 16 {
description "session A tries to discard-changes,
candidate is locked by B so this should fail"
session-name session-A
result-type error
result-error-tag in-use
command "discard-changes"
}
step 17 {
description "session A tries to unlock candidate,
candidate is locked by B so this should fail"
session-name session-A
result-type error
result-error-tag in-use
command "unlock target=candidate"
}
step 18 {
description "session B tries to unlock running,
which is locked by A so this should fail"
session-name session-B
result-type error
result-error-tag in-use
command "unlock target=running"
}
step 19 {
description "session B unlocks candidate"
session-name session-B
result-type ok
command "unlock target=candidate"
}
step 20 {
description "session A unlocks candidate"
session-name session-A
result-type ok
command "unlock target=running"
}
step 21 {
description "session A commits nothing due to
discard-changes from B"
session-name session-A
result-type ok
command "commit"
}
}
}
}