Automation Testing for NETCONF & YANG

Testing the Device NETCONF/YANG Interface

Sync Device Configuration into NSO

To synchronize configuration data from the device into NSO, use the following NSO CLI command:

Terminal
Copyadmin@ncs# devices device test1a sync-from
result true

Configure the device using NSO

To configure the device using the NETCONF NED and see the resulting device configuration, use the following NSO CLI commands:

Terminal
Copyadmin@ncs(config)# devices device test1a config dhcpd:dhcp maxLeaseTime 6000s
admin@ncs(config-config)# commit
Commit complete.
admin@ncs(config-config)# exit
admin@ncs(config-device-test1a)# exit
admin@ncs(config)# show full-configuration devices device test1a config
devices device test1a
 config
  dhcpd:dhcp maxLeaseTime 6000s
 !
!
admin@ncs(config)#

Test Transactionality using DrNED Examiner

DrNED Examiner can be used to test that a device supports the proper transactional behavior. Among other things, transactional behavior means that a device can accept any valid configuration regardless of the state it is in currently. For all transactions, it must also be true that the device configuration becomes exactly the configuration as modified by the transaction. In other words, it's not acceptable that a device modifies any other parts of the configuration other than the ones touched by the transaction. Note that a different device from the previous section is being used to demonstrate the DrNED Examiner tool in this section.

DrNED Examiner can test the conformance with these rules automatically but requires a smart selection of input configurations in order to find something useful. This is how it is done.

The last setup step is to initialize DrNED Examiner:

Terminal
Copyadmin@ncs# config
admin@ncs(config)# devices device test1a
admin@ncs(config-device-test1a)# drned-xmnr setup setup-xmnr overwrite true

Configure the device to an interesting configuration state and make sure that NSO is in sync with the device. Changes may be entered directly on the device, followed by a "sync-from" in NSO. Alternatively, changes may be entered in NSO using "ncs_cli" and committed to the device. Once a configuration is running fine on the device and NSO is in sync, issue the operational command:

Terminal
Copydrned-xmnr state record-state state-name <state-name>

The state name needs to be a valid file name but can otherwise be chosen freely. The name will be used later to describe which configuration state transitions that have issues.

Keep doing this with interesting configurations for a while, so that you have at least 4 states recorded. You can list the names of the states you have recorded, or delete ones you don't want to keep using:

Terminal
Copydrned-xmnr state list-states
drned-xmnr state delete-state state-name <state-name>

Then, when enough states have been collected, DrNED Examiner can start testing that all transitions work automatically. By default, all transitions are tried. That should give a safe result. Since testing all-to-all configuration transitions grows exponentially with the number of states, it's also possible to limit the number of transitions to try out. The test will then not be conclusive but may be more reasonable to run. You can run the test in any of these ways:

Terminal
Copydrned-xmnr transitions explore-transitions
drned-xmnr transitions explore-transitions stop-after { seconds 30 }
drned-xmnr transitions explore-transitions stop-after { minutes 5 }
drned-xmnr transitions explore-transitions stop-after { hours 12 }
drned-xmnr transitions explore-transitions stop-after { days 2 }
drned-xmnr transitions explore-transitions stop-after { cases 20 }
drned-xmnr transitions explore-transitions stop-after { percent 10 }

The sequence of transitions to try is selected randomly. Two different runs will therefore not yield the same test pattern.

A test run might look like this:

Terminal
Copyadmin@ncs(config-device-test1a)# drned-xmnr transitions explore-transitions stop-after { percent 10 }
Found 8 states recorded for device test1a which gives a total of 56 transitions.

Starting from known state noloop8
... failed setting known state

Starting from known state bundle-vrf
Transition 1/56: bundle-vrf ==> bundle-vrf-ipv4
Transition 2/56: bundle-vrf-ipv4 ==> bundle-vrf-ipv46
Transition 3/56: bundle-vrf-ipv46 ==> bundle-vrf-ipv4
Transition 4/56: bundle-vrf-ipv4 ==> bundle-vrf
Transition 5/56: bundle-vrf ==> bundle-vrf-ipv46
Transition 6/56: bundle-vrf-ipv46 ==> bundle-vrf
Requested stop-after time limit reached
success Completed successfully

The comment "...failed setting known state" is an indication that something isn't right. But since DrNED Examiner didn't know exactly what state the device was in before we started (or after a transaction has failed) it's hard to give instructions on how to repeat the issue, so this problem is ignored.

If an issue is found, the outcome might look like this:

Terminal
Copyadmin@ncs(config-device-test1a)# drned-xmnr transitions explore-transitions stop-after { percent 10 }
Found 8 states recorded for device test1a which gives a total of 56 transitions.

Starting from known state bundle-vrf-ipv4
Transition 1/56: bundle-vrf-ipv4 ==> bundle-vrf-ipv46
Transition 2/56: bundle-vrf-ipv46 ==> bundle-vrf-ipv4
Transition 3/56: bundle-vrf-ipv4 ==> bundle-vrf
Transition 4/56: bundle-vrf ==> bundle-vrf-ipv4
Transition 5/56: bundle-vrf-ipv4 ==> initial
   transaction-failed

Starting from known state no-bundle-ipv46
Transition 6/56: no-bundle-ipv46 ==> bundle-vrf-ipv4
Requested stop-after coverage limit reached
failure transaction-failed: bundle-vrf-ipv4 ==> initial

Here, the transition from state bundle-vrf-ipv4 to initial failed. transaction-failed means the device refused to go between these two states. If you see out-of-sync, that means the transaction was accepted by the device, but the configuration of the device is different than expected. Usually this means the device created/changed some other values than the ones specified in the transaction. In any case, the device configuration isn't what it should be. Assuming NETCONF tracing was enabled, this failure can now be debugged by manually invoking the configuration states in question:

Terminal
Copyadmin@ncs(config-device-test1a)# drned-xmnr transitions transition-to-state state-name bundle-vrf-ipv4
success Done
admin@ncs(config-device-test1a)# drned-xmnr state transition-to-state state-name initial        
failure transaction-failed
admin@ncs# file show logs/netconf-test1a.trace
...
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
     message-id="4">
  <edit-config xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
    <target>
      <candidate/>
    </target>
    <test-option>test-then-set</test-option>
    <error-option>rollback-on-error</error-option>
    <config>
      <interface-configurations xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-ifmgr-cfg">
        <interface-configuration>
...
    <error-message xml:lang="en">'RSI' detected the 'fatal' condition 'The interface's numbered and unnumbered IPv4/IPv6 addresses must be removed prior to changing or deleting the VRF'</error-message>
  </rpc-error>
</rpc-reply>

Clearly, a violation of the transactional behavior. The device may also have log files worth looking at to understand the issue.

Minimum Success Criteria for demonstrating NSO Interoperability

For demonstrating NSO interoperability, there is no need to aim for all supported YANG models by the device. It should start with the most typical use cases of the device and only a subset of the YANG models. Additional use cases can then be added over time to cover more device features.

The following are the suggested steps for achieving the minimum success criteria for NSO interoperability:

  1. Select one or more common use cases for your product.
  2. Define a typical base configuration (day-0 configuration) as a NETCONF test vector.
  3. Define one or more day-1 running configurations as NETCONF test vectors for your product that can be used to demonstrate each of your use cases.
  4. Define a couple more intermediate incremental configurations as NETCONF test vectors that will partially set up your system towards the desired day-1 configuration for each use case.
  5. For each use case, there should be 4 or more NETCONF test vectors as defined from steps 2 through 4.
  6. Use the DrNED Examiner tool to demonstrate that all possible combinations of state transitions for the identified NETCONF test vectors can be achieved successfully.
Next