The NSO Python VM

NSO is capable of starting one or several Python VMs where Python code in user provided packages can run.

An NSO package containing a python directory will be considered to be a Python Package. By default, a Python VM will be started for each Python package that has got a python-class-name defined in its package-meta-data.xml file. In this Python VM the PYTHONPATH environment variable will be pointing to the python directory in the package.

If any required-package that is listed in the package-meta-data.xml contains a python directory, the PATH to that directory will be added to the PYTHONPATH of the started Python VM and thus its accompanying Python code will be accessible.

Several Python packages can be started in the same Python VM if their corresponding package-meta-data.xml files contains the same python-package/vm-name.

A Python package skeleton can be created by making use of the ncs-make-package command:

ncs-make-package --service-skeleton python <package-name>

The tailf-ncs-python-vm.yang defines the python-vm container which, along with ncs.conf, is the entry point for controlling the NSO Python VM functionality. Study the content of the YANG model in Example 26, “The Python VM YANG model”. For a full explanation of all the configuration data, look at the YANG file and man ncs.conf. Here will follow a description of the most important configuration parameters.

Note that some of the nodes beneath python-vm are by default invisible due to a hidden attribute. In order to make everything beneath python-vm visible in the CLI, two steps are required. First the following XML snippet must be added to ncs.conf:

<hide-group>
  <name>debug</name>
</hide-group>

Now the unhide command may be used in the CLI session:

admin@ncs(config)# unhide debug
admin@ncs(config)#

The sanity-checks/self-assign-warning control the self-assignment warnings for Python services with off, log, and alarm (default) modes. An example of a self-assignment:

class ServiceCallbacks(Service):
    @Service.create
    def cb_create(self, tctx, root, service, proplist):
        self.counter = 42

As several service invocations may run in parallel, self-assignment will likely cause difficult to debug issues. An alarm or a log entry will contain a warning and a keypath to the service instance that caused the warning. Example log entry:

<WARNING> ... Assigning to self is not thread safe: /mysrvc:mysrvc{2}

With the logging/level the amount of logged information can be controlled. This is a global setting applied to all started Python VMs unless explicitly set for a particular VM, see the section called “Debugging of Python packages”. The levels corresponds to the pre-defined Python levels in the Python logging module, ranging from level-critical to level-debug.

The logging/log-file-prefix define the prefix part of the log file path used for the Python VMs. This prefix will be appended with a Python VM specific suffix which is based on the Python package name or the python-package/vm-name from the package-meta-data.xml file. The default prefix is logs/ncs-python-vm so e.g. if a Python package named l3vpn is started, a logfile with the name logs/ncs-python-vm-l3vpn.log will be created.

The status/start and status/current contains operational data. The status/start command will show information about what Python classes, as declared in the package-meta-data.xml file, that where started and whether the outcome was successful or not. The status/current command will show which Python classes that are currently running in a separate thread. The latter assume that the user provided code cooperate by informing NSO about any thread(s) started by the user code, see the section called “Structure of the User provided code”.

The start and stop actions makes it possible to start and stop a particular Python VM.

          
> yanger -f tree tailf-ncs-python-vm.yang
          
submodule: tailf-ncs-python-vm (belongs-to tailf-ncs)
  +--rw python-vm
     +--rw sanity-checks
     |  +--rw self-assign-warning?   enumeration
     +--rw logging
     |  +--rw log-file-prefix?   string
     |  +--rw level?             py-log-level-type
     |  +--rw vm-levels* [node-id]
     |     +--rw node-id    string
     |     +--rw level      py-log-level-type
     +--rw status
     |  +--ro start* [node-id]
     |  |  +--ro node-id     string
     |  |  +--ro packages* [package-name]
     |  |     +--ro package-name    string
     |  |     +--ro components* [component-name]
     |  |        +--ro component-name    string
     |  |        +--ro class-name?       string
     |  |        +--ro status?           enumeration
     |  |        +--ro error-info?       string
     |  +--ro current* [node-id]
     |     +--ro node-id     string
     |     +--ro packages* [package-name]
     |        +--ro package-name    string
     |        +--ro components* [component-name]
     |           +--ro component-name    string
     |           +--ro class-names* [class-name]
     |              +--ro class-name    string
     |              +--ro status?       enumeration
     +---x stop
     |  +---w input
     |  |  +---w name    string
     |  +--ro output
     |     +--ro result?   string
     +---x start
        +---w input
        |  +---w name    string
        +--ro output
           +--ro result?   string

The package-meta-data.xml file must contain a component of type application with a python-class-name specified as shown in the example below.

<component>
  <name>L3VPN Service</name>
  <application>
    <python-class-name>l3vpn.service.Service</python-class-name>
  </application>
</component>
<component>
  <name>L3VPN Service model upgrade</name>
  <upgrade>
    <python-class-name>l3vpn.upgrade.Upgrade</python-class-name>
  </upgrade>
</component>

The component name (L3VPN Service in the example) is a human readable name of this application component. It will be shown when doing show python-vm in the CLI. The python-class-name should specify the Python class that implements the application entry point. Note that it needs to be specified using Python's dot-notation and should be fully qualified (given the fact that PYTHONPATH is pointing to the package python directory).

Study the excerpt of the directory listing from a package named l3vpn below.

packages/
+-- l3vpn/
    +-- package-meta-data.xml
    +-- python/
    |   +-- l3vpn/
    |       +-- __init__.py
    |       +-- service.py
    |       +-- upgrade.py
    |       +-- _namespaces/
    |           +-- __init__.py
    |           +-- l3vpn_ns.py
    +-- src
        +-- Makefile
        +-- yang/
            +-- l3vpn.yang

Look closely at the python directory above. Note that directly under this directory is another directory named as the package (l3vpn) that contains the user code. This is an important structural choice which eliminates the chance of code clashes between dependent packages (only if all dependent packages uses this pattern of course).

As you can see, the service.py is located according to the description above. There is also a __init__.py (which is empty) there to make the l3vpn directory considered a module from Python's perspective.

Note the _namespaces/l3vpn_ns.py file. It is generated from the l3vpn.yang model using the ncsc --emit-python command and contains constants representing the namespace and the various components of the YANG model, which the User code can import and make use of.

The service.py file should include a class definition named Service which acts as the component's entry point. See the section called “The application component” for details.

Notice that there is also a file named upgrade.py present which hold the implementation of the upgrade component specified in the package-meta-data.xml excerpt above. See the section called “The upgrade component” for details regarding upgrade components.

import ncs

class Service(ncs.application.Application):
    def setup(self):
        # The application class sets up logging for us. It is accessible
        # through 'self.log' and is a ncs.log.Log instance.
        self.log.info('Service RUNNING')

        # Service callbacks require a registration for a 'service point',
        # as specified in the corresponding data model.
        #
        self.register_service('l3vpn-servicepoint', ServiceCallbacks)

        # If we registered any callback(s) above, the Application class
        # took care of creating a daemon (related to the service/action point).

        # When this setup method is finished, all registrations are
        # considered done and the application is 'started'.

    def teardown(self):
        # When the application is finished (which would happen if NCS went
        # down, packages were reloaded or some error occurred) this teardown
        # method will be called.

        self.log.info('Service FINISHED')

import ncs
import _ncs


class Upgrade(ncs.upgrade.Upgrade):
    """An upgrade 'class' that will be instantiated by NSO.

    This class can be named anything as long as NSO can find it using the
    information specified in <python-class-name> for the <upgrade>
    component in package-meta-data.xml.

    Is should inherit ncs.upgrade.Upgrade.

    NSO will instantiate this class using the empty contructor.
    The class MUST have a method named 'upgrade' (as in the example below)
    which will be called by NSO.
    """

    def upgrade(self, cdbsock, trans):
        """The upgrade 'method' that will be called by NSO.

        Arguments:
        cdbsock -- a connected CDB data socket for reading current (old) data.
        trans -- a ncs.maapi.Transaction instance connected to the init
                 transaction for writing (new) data.

        There is no need to connect a CDB data socket to NSO - that part is
        already taken care of and the socket is passed in the first argument
        'cdbsock'. A session against the DB needs to be started though. The
        session doesn't need to be ended and the socket doesn't need to be
        closed - NSO will do that automatically.

        The second argument 'trans' is already attached to the init transaction
        and ready to be used for writing the changes. It can be used to create a
        maagic object if that is preferred. There's no need to detach or finish
        the transaction, and, remember to NOT apply() the transaction when work
        is finished.

        The method should return True (or None, which means that a return
        statement is not needed) if everything was OK.
        If something went wrong the method should return False or throw an
        error. The northbound client initiating the upgrade will be alerted
        with an error message.

        Anything written to stdout/stderr will end up in the general log file
        for spurious output from Python VMs. If not configured the file will
        be named ncs-python-vm.log.
        """

        # start a session against running
        _ncs.cdb.start_session2(cdbsock, ncs.cdb.RUNNING,
                                ncs.cdb.LOCK_SESSION | ncs.cdb.LOCK_WAIT)

        # loop over a list and do some work
        num = _ncs.cdb.num_instances(cdbsock, '/path/to/list')
        for i in range(0, num):
            # read the key (which in this example is 'name') as a ncs.Value
            value = _ncs.cdb.get(cdbsock, '/path/to/list[{0}]/name'.format(i))
            # create a mandatory leaf 'level' (enum - low, normal, high)
            key = str(value)
            trans.set_elem('normal', '/path/to/list{{{0}}}/level'.format(key))

        # not really needed
        return True

        # Error return example:
        #
        # This indicates a failure and the string written to stdout below will
        # written to the general log file for spurious output from Python VMs.
        #
        # print('Error: not implemented yet')
        # return False

Python code packages are not running with an attached console and the standard out from the Python VMs are collected and put into the common log file ncs-python-vm.log. Possible python compilation errors will also end up in this file.

Normally the logging objects provided by the Python APIs is used. They are based on the standard Python logging module. This gives the possibility to control the logging if needed, e.g. getting a module local logger to increase logging granularity.

The default logging level is set to info. For debugging purposes it is very useful to increase the logging level:

    $ ncs_cli -u admin
    admin@ncs> config
    admin@ncs% set python-vm logging level level-debug
    admin@ncs% commit

This sets the global logging level and will affect all started Python VMs. It is also possible to set the logging level for a single package (or multiple packages running in the same VM), which will take precedence over the global setting:

    $ ncs_cli -u admin
    admin@ncs> config
    admin@ncs% set python-vm logging vm-levels pkg_name level level-debug
    admin@ncs% commit

The debugging output are printed to separate files for each package and the log file naming is ncs-python-vm-pkg_name.log

Log file output example for package l3vpn:

    $ tail -f logs/ncs-python-vm-l3vpn.log
    2016-04-13 11:24:07 - l3vpn - DEBUG - Waiting for Json msgs
    2016-04-13 11:26:09 - l3vpn - INFO - action name: double
    2016-04-13 11:26:09 - l3vpn - INFO - action input.number: 21

There are occasions where the standard Python installation is incompatible or maybe not preferred to be used together with NSO. In such cases there are several options to tell NSO to use another Python installation for starting a Python VM.

By default NSO will use the file $NCS_DIR/bin/ncs-start-python-vm when starting a new Python VM. The last few lines in that file reads:

        if [ -x "$(which python3)" ]; then
            echo "Starting python3 -u $main $*"
            exec python3 -u "$main" "$@"
        fi
        echo "Starting python -u $main $*"
        exec python -u "$main" "$@"

As seen above NSO first looks for python3 and if found it will be used to start the VM. If python3 is not found NSO will try to use the command python instead. Here we describe a couple of options for deciding which Python NSO should start.

$ cd $NCS_DIR/bin
$ pwd
/usr/local/nso/bin
$ cp ncs-start-python-vm my-start-python-vm
$ # Use your favourite editor to update the last lines of the new
$ # file to start the desired Python executable.

<python-vm>
    <start-command>/usr/local/nso/bin/my-start-python-vm</start-command>
</python-vm>