DrNED Examiner

A Doctor's bag of tools for examining & diagnosing your NSO NEDs

Purpose

DrNED Examiner helps you with testing your NED or device using the DrNED
package. Using the tool's actions you can:

• set up the environment (or at least get help with that)
• save/import device configurations (or states) to be used later in tests
• invoke DrNED tests that verify if the device transitions between those
  states smoothly
• make DrNED generate reports on how well the state transitions cover the
  device model
• log tools allow reviewing the communication between NSO and the device,
  debugging situations when the device configuration violates the device's
  YANG contract, and when testing transactionality properties

Prerequisites

The tool is supported to run on Python 3.6 or newer.

Apart form that, several Python packages are needed: pytest, pexpect, and
lxml; pytest version needs to be at least 3.0. There are several options
how to install these packages:

1. Quite likely, they are available in your system distribution repositories,
   for example in Ubuntu they can be installed like

    $ sudo apt install python3-pytest python3-lxml python3-pexpect
   

   Note that these are packages that run with Python 3. If you happen to be
   restricted to Python 2, install python-pytest, python-lxm, and
   python-pexpect.

   Care must be taken though if the pytest version meets requirements,
   namely on Ubuntu 16.04 it does not:

    $ apt show python3-pytest
    ...
    Version: 2.8.7-4
   

2. All three packages are published through the Python Package Inventory and
   can be installed from there; it requires that pip is installed (on Ubuntu
   as python-pip). The requirements are enumerated in requirements.txt
   file that can be provided to pip:

    $ pip install --user -r requirements.txt
   

   or for a system-wide installation

    $ sudo pip install -r requirements.txt
   

3. If it is required that the packages are not only user-specific but also
   project-specific, it is possible to use so called python virtual
   environments. Using them can be greatly simplified by the tool pipenv
   which needs to be installed first (globally or per user):

   $ pip install --user pipenv
   

   or

   $ sudo pip install pipenv
   

   Have a look at the file Pipfile - it tells pipenv what packages and in
   what versions need to be installed for the project environment. The
   environment can be created with required packages using

   $ pipenv install
   

   Now, so as to enter the environment, run

   $ pipenv shell
   

   This starts a new OS shell with paths pointing to the installed packages.
   NSO or at least its Python VM should be started from this shell (see also
   the command pipenv run for an alternative method).

DrNED Examiner and all its prerequisites is installed in this
NSO-interoperability testing image.

What it can do

Apart from this file that presents an overview of the tool's capabilities,
the main documentation is the drned-xmnr.yang file, where each command is
described.

Before working with this tool you need a device configured in NSO and
connected. For doing that you may want to use the
Pioneer tool.

DrNED Examiner has two configuration parameters: /drned-xmnr/xmnr-directory
and /drned-xmnr/drned-directory. The former sets the location where the tool
stores its data and defaults to /tmp/xmnr; the latter default to the built-in
DrNED version that come with this NSO package. For using a different version of
DrNED, you can point to a different DrNED installation directory than the default
one. (As an alternative, set the environment variable DRNED before starting
NSO and configure the drned-directory to env)

The following sections briefly describe actions implemented by the tool. All
actions have only a textual output with fields success (populated if the
action completed successfully), failure and error (the latter may contain
more details about the problem in case of failure).

• Setup

  Run the action drned-xmnr/setup/setup-xmnr to prepare and populate the data
  directory. This is needed as the first action before any further work with the
  tool, or to clean up the directory (with parameter overwrite).

• Device states

  Once set up, you need to prepare device states. A device state is pretty much
  a device configuration, so you can configure the device and record its state,
  or you can directly import saved device configurations from various formats.

• Transitions

  The main purpose of this tool is to help you verify that the device under test
  transitions smoothly between states - you can move from state to state, or walk
  several states at once, or explore all possible transitions between several
  states. The tool invokes DrNED for all these tasks and uses its capabilities
  to detect auto-configuration issues, problems with rollback, etc.

• Coverage

  DrNED is capable of reporting how big part of the device model your tests have
  covered; the tool implements a simple wrapper around this DrNED capability,
  including status data providing the coverage report in a structured form.

  Typical usage:

  • reset coverage data:

      ncs(config-device-testdev)# drned-xmnr coverage reset
    

  • do transitions, i.e. walk-states, explore-states etc.

  • collect coverage data:

      ncs(config-device-testdev)# drned-xmnr coverage collect
    

    Note that in this short form the tool looks up the NED package and uses
    all its YANG modules. This may take long and yield very low coverage
    percentage. If only a subset of YANG modules is relevant for the
    coverage test, use a path pattern (or patterns) with wildcards pointing
    to the interesting files like this:

        ... coverage collect yang-patterns [ <pattern1> <pattern2> ... ]
    

  • inspect collected coverage data:

      ncs(config-device-testdev)# do show devices device testdev drned-xmnr coverage
      drned-xmnr coverage data nodes-total 13
      drned-xmnr coverage data lists-total 3
      drned-xmnr coverage data percents nodes read-or-set total 4
      ...
    

  The coverage data is available in the form of operational data until the
  next reset action.

Debugging issues

Your main tools for debugging issues are logs.

• First you have the DrNED log that you control the filtering of via
  /drned-xmnr/log-detail/cli. You can point /drned-xmnr/cli-log-file to
  a file where all CLI output will be stored. In an automated test
  environment, you would normally set the CLI detail to drned-overview to
  get the DrNED reports. When debugging issues, you may want to output
  everything to your CLI console with the log-detail set to all.
• Equally useful is the ncs-python-vm-drned-xmnr.log that you control through
  /python-vm/logging/vm-levels/drned-xmnr/level/. To maximize the information
  captured by this log, set to level-debug. The log can be found in the
  ncs-python-vm-drned-xmnr.log.
• It can often be helpful to look into the detailed trace of the communication
  with the (virtual/physical) device. Set the /devices/device/<my-device>/trace
  to pretty or raw to capture the communication between the NED and the device.
• For other issues detected and reported by NSO, refer to the NSO administration
  guide for troubleshooting adivce.

You can use device specific action parse-log-errors below device's drned-xmnr
node to parse either global drned-xmnr logfile (ncs-python-vm-drned-xmnr.log),
or device specific trace log.

The action reads the log file, and "greps" all the clusters of lines matching
an "error-paterns" defined in CDB under the path /drned-xmnr/error-patterns.
Here are few examples of typical errors that you may encounted in log file:

admin@ncs(config)# show full-configuration drned-xmnr error-patterns
drned-xmnr error-patterns match " !! "
 terminator ">>>> \"exit\" >>>>"
 max-lines  100
!
drned-xmnr error-patterns match "Aborted: RPC error"
 exact
!
drned-xmnr error-patterns match "E  "
 exact
!
drned-xmnr error-patterns match "sync-from failed:"
 exact
!

After executing the action, you can try checking the records retrieved from
the filesystem log files. Invoke the show command (use device name specific
for your use-case): show devices device MyDevice drned-xmnr parsed-problems.
Invoke the above sohw command with extra parameters for more in-depth details
(see tab completion in CLI).

Common problems

DrNED uses py.test as a tool for running tests. py.test cannot process
correctly command line arguments containing hyphens (-); DrNED works around
that by translating hyphens to tildes (~) in file names (but not in device
names), so it should be possible to use hyphens in state names (again: not in
device names), but due to the workaround, it is not possible to use tildes
there.

Testing the testing tool

If you intend to modify or extend the tool, you may want to test your changes -
please refer to the test README; note that the tests add few
additional prerequisities, see there.