Reasons for Automation Testing for NETCONF & YANG

In this section, we will briefly discuss some of the possible issues that can be caught during Automation Testing for NETCONF & YANG (NYAT).

Potential issues that can be found during Automation Testing for NETCONF & YANG include simple things such as data type mismatches between a leaf node's definition in the YANG data model and the NETCONF implementation. For example, enumerated data types defined in YANG may return incorrectly as integer values through NETCONF. Leafref data types may also return invalid or non-existing paths to your data model.

Other than data type mismatches, there may be other issues that are slightly more difficult to uncover without the use of the proper NETCONF testing tool. They include out-of-band configuration changes that happen with NETCONF’s operation. Some people call this auto-configs or self-modifying configuration. These are additional changes to the device’s configuration beyond what is specified in the operation. This issue will cause the configuration on your device to get out-of-sync with the NETCONF manager (client) unless special code is put in place after the operation to perform the synchronization. This causes unnecessary complexity and performance overhead for configuration orchestration software such as NSO.

Another issue that has to do with NETCONF transactions should also be tested and addressed on the device during the development cycle of your device’s software. This issue has to do with the order dependencies of the configuration changes. It is very common for legacy CLI-based devices to require configuration changes to be entered in a particular order. There can’t be order dependencies with NETCONF transactions. Relying on this order dependency will break the transactionality property of your NETCONF device. This will take away the simplicity that configuration orchestration software relies on to be able to consistently and reliably go back and forth between different configuration states by using simple transactional operations without having to know the order of how configuration changes should happen on the device.

How your device deals with default values for the NETCONF / operations is also important. If your device isn’t implemented according to the "With-defaults Capability for NETCONF", as described in RFC 6243, this issue can also be caught during this interoperability testing.

These are a few examples of the types of issues which could arise without performing Automation Testing for NETCONF & YANG to catch them.

You can define test cases with the tools as described in this document according to the most typical use cases of your device in order to detect data type mismatches, validate its transactionality behavior, that it is free of side effects from a NETCONF operation, as well as other test cases of your own design.