Automated Testing

Automated regression testing is provided using test-suite templates to describe actions and expected responses and results. The test-suite command is used to access this feature.

Tests rely on named sessions in order to send commands from different sessions. Each test step can be directed at a different session. The “locking” example in a later section shows how test step commands can be either expected to succeed (e.g., first session attempts <lock> command) and fail (e.g., second session attempts <lock> but the datastore is already locked).

A test suite that only needs to send commands to the current session does not need to use named sessions. The session connection and shutdown steps do not need to be done in the test-suite.

The start-session command is usually used within the 'setup' section for each session required in all of the tests within the test-suite. The server must already be running.

The 'setup' and 'cleanup' sections defined within a test suite are called 'CLI text blocks'. They use a special configuration file syntax which allows maximum flexibility for the test designer. These sections begin and and with square brackets ('[' and ']') instead of angle brackets ('{' and '}'). There are no leafs within these containers. Instead they contain plain command lines exactly like a script.

Test Templates

The test suite file structure is shown below:

../_images/test-suite-diagram-cropped.png
  • test-suite-file: collection of test-suites in a single file. Multiple files can be loaded. If the --autotest CLI parameter is 'true' (default) then yangcli-pro will look for the default test file, called $HOME/.yumapro/yangcli_pro_tests.conf .

  • test-suite: collection of tests. Each test-suite has an optional 'setup' section which is executed once before the tests are run, and an optional 'cleanup' section that is run after all the tests have run

  • test: collection of steps to perform a specific test

  • step: a command to be executed locally or remotely to a specified session. Each step for a remote command can be configured to check the server response received.

YANG File Defining Test Templates

module yumaworks-test {

  namespace "http://yumaworks.com/ns/yumaworks-test";

  prefix "ywtest";

  import yuma-types { prefix yt; }
  import yumaworks-extensions { prefix ywx; }

  organization "YumaWorks, Inc.";

  contact
    "Support <[email protected]>";

  description
    "This module contains data structures representing
     replayable test suites for specific use cases and YANG modules

        +---------------------+
      +-+------------------+  |
      |                    |  |
      |     Test Suite     |  |
      |                    |--+
      +--------------------+
       |          |
       |          |
       |          V
       |    +---------------------+
       |  +-+------------------+  |
       |  |                    |  |
       |  |        Test        |  |
       |  |                    |--+
       |  +--------------------+
       |          |
       |          |
       |          V
       |      +---------------------+
       |    +-+------------------+  |
       |    |                    |  |
       |    |        Step        |  |
       |    |                    |--+
       |    +--------------------+
       |
       V
      +---------------------+
    +-+------------------+  |
    |                    |  |
    |      Servers       |  |
    |                    |--+
    +--------------------+
              |
              |
              V
        +---------------------+
      +-+------------------+  |
      |                    |  |
      |      Sessions      |  |
      |                    |--+
      +--------------------+


     Copyright (c) 2013 - 2023 YumaWorks, Inc. All rights reserved.

     Redistribution and use in source and binary forms, with or
     without modification, is permitted pursuant to, and subject
     to the license terms contained in, the BSD 3-Clause License
     http://opensource.org/licenses/BSD-3-Clause";

  revision 2023-03-16 {
    description
       "Add message-indent parameter";
  }

  revision 2013-03-31 {
    description
       "Rename to yumaworks-test";
  }

  revision 2013-01-21 {
    description
       "Initial version";
  }

  typedef response-type {
    type enumeration {
      enum none {
        description "Local command, no <rpc-reply> expected.";
      }
      enum ok {
        description "Expecting the <ok> reply.";
      }
      enum data {
        description "Expecting a data reply.";
      }
      enum error {
        description "Expecting an rpc-error reply.";
      }
    }
    description
      "The type of response expected from the server
       for this request.";
  }

  container test-suites {

     list test-suite {

      key name;

      leaf name {
        type yt:NcxName;
         description
          "The test suite name.";
      }

      leaf description {
        type string;
        description "Description of this test suite.";
      }

      container setup {
        ywx:cli-text-block;
        description "Test suite setup commands";
      }

      container cleanup {
        ywx:cli-text-block;
        description "Test suite cleanup commands";
      }

      leaf-list run-test {
        ordered-by user;
        type yt:NcxName;
        min-elements 1;
        description
          "The ordered list of test names to run in this test suite.
           At least 1 test must be specified.";
      }

      leaf message-indent {
        type int8 {
          range "-1 .. 9";
        }
        default 2;
        description "message-indent used for the test.";
      }

      list test {
        description
          "The unit-tests that are configured to be run.
           At least 1 test must be configured.";
        ordered-by user;
        min-elements 1;
        key name;

        leaf name {
          type yt:NcxName;
          description "The name of this unit test.";
        }

        leaf description {
          type string;
          description "Description of this unit test.";
        }

        leaf-list must-pass {
          type yt:NcxName;
          description
           "The names of the tests that have already been
            run and pased for this test to be attempted,
            The test will be skipped if any test in the
            must-pass list has been attempted and the
            test failed.

            If the named test has not been run yet
            this test will fail and be skipped.  If the named
            test was skipped, then it will not cause this test
            to be skipped, only if it did not run at all or
            if it ran and passed.";
        }

        list step {
          description
            "A list of test steps to be done in order.";
          ordered-by user;
          key name;

          leaf name {
            type string {
              length "1 .. 64";
            }
            description "The name of this unit test step.";
          }

          leaf description {
            type string;
            description "Description of this unit test step.";
          }

          leaf session-name {
            type yt:NcxName;
            description
             "The name of the session to use.
              Empty if the test session should be used";
          }

          leaf result-type {
            type response-type;
            description
              "The expected response type. If this leaf is
               missing then any response type is acceptable.";
          }

          leaf result-error-tag {
            when "../result-type = 'error'";
            type string;
            description
              "The error-tag value expected if the result-type
               is 'error'.  If not set, then any error-tag
               value is acceptable.";
          }

          leaf result-error-apptag {
            when "../result-type = 'error'";
            type string;
            description
              "The error-app-tag value expected if the result-type
               is 'error'.  If not set, then any error-app-tag
               value is acceptable.";
          }

          leaf-list result-error-info {
            when "../result-type = 'error'";
            type yt:NcxName;
            description
              "The error-info element name expected if the result-type
               is 'error'.";
          }

          leaf command {
            type string;
            mandatory true;
            description "The yangcli command line string to use";
          }

          anyxml rpc-reply-data {
            when "../result-type = 'data'";
            description
              "The contents of <rpc-reply> element that are
                expected to be returned in the reply.
                This element itself represents <rpc-reply>
                and any child nodes are the nodes returned
                by the server.";
           }

         }  // list step
      }     // list test
    } // list test-suite
  } // container test-suites


}

RPC Reply Testing

Each step has a “response-type” field that specifies what kind of RPC reply testing is expected.

  • none: Do not perform any RPC reply testing, or this step does not send an RPC request to a server

  • ok: An RPC reply is expected with an <ok> response

  • data: An RPC reply is expected with some sort of data response. The expected response is stored in an XML file and compared to the actual RPC reply when the step is executed.

    • Initial files are created automatically during the “record-test” procedure.

    • The files are stored in the $HOME/.yumapro/recordtest directory.

    • Each filename has the form <suite-name>_<test-name>_<step-number>_record.xml

    • During the test a temporary file will be created for each data file to save the RPC reply. These files will be deleted when the test suite is completed.

    • Each file contains a single <rpc-reply> element.

    • These files can be edited, created, by hand, or even deleted.

    • Wildcard fields can be used in this file to allow any value to match, removing this node from the comparison test.

  • error: An RPC reply is expected with an <rpc-error> response

    • error-tag: The expected error-tag in the <rpc-error> response

    • error-app-tag: The expected error-app-tag in the <rpc-error> response

    • result-error-info: A list of names of elements expected in the <error-info> container within the <rpc-error> response

Using Wildcards in Data Response Testing

It is possible simply test for a node within the RPC reply instead of comparing its actual value. Some data such as timestamps and counters may change on every retrieval. Such a node can cause the entire step to fail when comparing the <rpc-reply> contents.

The wildcard string is a sequence of 5 characters (underscore, asterisk, asterisk, asterisk, underscore)

_***_

If this string is used as the value for an element, then that node will be skipped in the comparison.

Example Step:

  • Suite: s1

  • Test: t1

  • Step: 2

  • Command: sget /netconf-state/datastores

  • File: $HOME/.yumapro/recordtest/s1_t1_2_record.xml

Original Data File Created During record-test:

<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <data>
    <netconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
      <datastores>
        <datastore>
          <name>candidate</name>
          <last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">2019-12-05T23:21:27Z</last-modified>
        </datastore>
        <datastore>
          <name>running</name>
          <last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">2019-12-05T23:21:27Z</last-modified>
        </datastore>
      </datastores>
    </netconf-state>
  </data>
</rpc-reply>

Altered Data File Wild Wildcard Strings:

<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <data>
    <netconf-state xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
      <datastores>
        <datastore>
          <name>candidate</name>
          <last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">_***_</last-modified>
        </datastore>
        <datastore>
          <name>running</name>
          <last-modified xmlns="http://netconfcentral.org/ns/yuma-time-filter">_***_</last-modified>
        </datastore>
      </datastores>
    </netconf-state>
  </data>
</rpc-reply>

Example Test File

test-suites {
   test-suite first-test {
     description "A set of tests to validate NETCONF locking is working
          correctly."

     setup [
      # example variables
       $$server ="test1"
       $$admin = "andy"
       $$user1 = "andy"
       $$pass1 = "fredlow"
       $$user2 = "andy2"
       $$pass2 = "fredlow"
       $$testlog = "$HOME/tests/first-test.log"
       $$testmod = "test"

       # 2 sessions used in this test; server must already be running
       start-session session-A
       start-session session-B
     ]

     cleanup [
       stop-session session-A
       stop-session session-B
     ]

     run-test locks

     test locks {

       description "Use 2 sessions to test global locks on the
          candidate and running datastores. Expects the server
          to already be started with --module=test
          and --access-control=off."

       step 1 {
         description "session A locks the running config;
           needs to lock candidate too, but does not!"
         session-name session-A
         result-type ok
         command "lock target=running"
       }

       step 2 {
         description "session B tries to lock the running config"
         session-name session-B
         result-type error
         result-error-tag lock-denied
         result-error-info session-id
         command "lock target=running"
       }

       step 3 {
         description "session A tries to write to the target config"
         session-name session-A
         result-type ok
         command "merge /uint8.1 value=10"
       }

       step 4 {
         description "session B tries to write to the target config
           candidate is not locked so this should work"
         session-name session-B
         result-type ok
         command "merge /uint8.1 value=12"
       }

       step 5 {
         description "session B tries to commit the candidate to running
           running is locked so this should fail"
         session-name session-B
         result-type error
         result-error-tag in-use
         command "commit"
       }

       step 6 {
         description "session A tries to lock the candidate config
           this should fail since the candidate is dirty"
         session-name session-A
         result-type error
         result-error-tag resource-denied
         command "lock target=candidate"
       }

       step 7 {
         description "session B tries to lock the candidate config
           this should fail since the candidate is dirty"
         session-name session-B
         result-type error
         result-error-tag resource-denied
         command "lock target=candidate"
       }

       step 8 {
         description "session B issues a discard-changes"
         session-name session-B
         result-type ok
         command "discard-changes"
       }

       step 9 {
         description "session A issues a discard-changes"
         session-name session-A
         result-type ok
         command "discard-changes"
       }

       step 10 {
         description "session B locks the candidate config"
         session-name session-B
         result-type ok
         command "lock target=candidate"
       }

       step 11 {
         description "session A tries to lock the candidate config,
           which should fail because it is already locked"
         session-name session-A
         result-type error
         result-error-tag lock-denied
         result-error-info session-id
         command "lock target=candidate"
       }

       step 12 {
         description "session A tries to write to the target config,
           which could fail because candidate is locked"
         session-name session-A
         result-type error
         result-error-tag in-use
         command "merge /uint8.1 value=10"
       }

       step 13 {
         description "session B tries to write to the target config"
         session-name session-B
         result-type ok
         command "merge /uint8.1 value=12"
       }

       step 14 {
         description "session A tries to commit the candidate to running,
           candidate is locked by B so this should fail"
         session-name session-A
         result-type error
         result-error-tag in-use
         command "commit"
       }

       step 15 {
         description "session B tries to commit the candidate to running,
           running is locked by A so this should fail"
         session-name session-B
         result-type error
         result-error-tag in-use
         command "commit"
       }

       step 16 {
         description "session A tries to discard-changes,
           candidate is locked by B so this should fail"
         session-name session-A
         result-type error
         result-error-tag in-use
         command "discard-changes"
       }

       step 17 {
         description "session A tries to unlock candidate,
           candidate is locked by B so this should fail"
         session-name session-A
         result-type error
         result-error-tag in-use
         command "unlock target=candidate"
       }

       step 18 {
         description "session B tries to unlock running,
           which is locked by A so this should fail"
         session-name session-B
         result-type error
         result-error-tag in-use
         command "unlock target=running"
       }

       step 19 {
         description "session B unlocks candidate"
         session-name session-B
         result-type ok
         command "unlock target=candidate"
       }

       step 20 {
         description "session A unlocks candidate"
         session-name session-A
         result-type ok
         command "unlock target=running"
       }

       step 21 {
         description "session A commits nothing due to
            discard-changes from B"
         session-name session-A
         result-type ok
         command "commit"
       }

     }
   }
}