Services Deep Dive

Each service type in NSO extends a part of the data model (a list or a container) with the ncs:servicepoint statement and the ncs:service-data grouping. This is what defines an NSO service.

The service point instructs NSO to involve the service machinery (Service Manager) for management of that part of the data tree and the ncs:service-data grouping contains definitions common to all services in NSO. Defined in tailf-ncs-services.yang, ncs:service-data includes parts that are required for proper operation of FASTMAP and the Service Manager. Every service must therefore use this grouping as part of its data model.

In addition, ncs:service-data provides a common service interface to the users, consisting of:

While not part of ncs:service-data as such, you may consider the service-commit-queue-event notification part of the core service interface. The notification provides information about the state of the service when service uses the commit queue. As an example, an event-driven application uses this notification to find out when a service instance has been deployed to the devices. See the showcase_rc.py script in examples.ncs/development-guide/concurrency-model/perf-stack/ for sample Python code, leveraging the notification. See tailf-ncs-services.yang for the full definition of the notification.

NSO Service Manager is responsible for providing the functionality of the common service interface, requiring no additional user code. This interface is the same for classic and nano services, whereas nano services further extend the model.

NSO calls into Service Manager when accessing actions and operational data under the common service interface, or when the service instance configuration data (the data under the service point) changes. NSO being a transactional system, configuration data changes happen in a transaction.

When applied, a transaction goes through multiple stages, as shown by the progress trace (e.g. using commit | details in the CLI). The detailed output breaks up the transaction into four distinct phases:

These phases deal with how the network-wide transactions work: validation phase prepares and validates the new configuration (including NSO copy of device configurations), then the CDB processes the changes and prepares them for local storage in the write-start phase. The prepare stage sends out the changes to network through the Device Manager and the HA system. The changes are staged (e.g. in candidate data store) and validated if device supports it, otherwise the changes are activated immediately. If all systems took the new configuration successfully, enter the commit phase, marking the new NSO configuration as active and activating or committing the staged configuration on remote devices. Otherwise, enter the abort phase, discarding changes and ask NEDs to revert activated changes on devices that do not support transactions (e.g. without candidate data store).

Figure 18. Typical transaction phases

There are also two types of locks involved with the transaction that are of interest to service developer; the service write lock and the transaction lock. The latter is a global lock, required to serialize transactions, while the former is a per-service-type lock for serializing services that cannot be run in parallel. See Scaling and Performance Optimization for more details and their impact on performance.

The first phase, historically called validation, does more than just validate data and is the phase a service deals with the most. The other three support the NSO service framework but a service developer rarely interacts with directly.

We can further break down the first phase into the following stages:

When transaction starts applying, NSO captures the initial intent and creates a rollback file, which allows one to reverse or roll back the intent. For example, the rollback file might contain the information that you changed a service instance parameter but it would not contain the service-produced device changes.

Then the first, partial validation takes place. It ensures the service input parameters are valid according to the service YANG model, so the service code can safely use provided parameter values.

Next, NSO runs transaction hooks and performs the necessary transforms, which alter the data before it is saved, for example encrypting passwords. This is also where Service Manager invokes FASTMAP and service mapping callbacks, recording the resulting changes. NSO takes service write locks in this stage, too.

After transforms, there are no more changes to the configuration data and the full validation starts, including YANG model constraints over the complete configuration, custom validation through validation points, and configuration policies (the section called “Policies” in User Guide).

Figure 19. Stages of transaction validation phase

Throughout the phase, transaction engine makes checkpoints, so it can restart the transaction faster in case of concurrency conflicts. The check for conflicts happens at the end of this first phase, when NSO also takes the global transaction lock. Concurrency is further discussed in NSO Concurrency Model.

The main callback associated with a service point is the create callback, designed to produce the required (new) configuration, while FASTMAP takes care of the other operations, such as update and delete.

NSO implements two additional, optional callbacks for scenarios where create is insufficient. These are pre- and post-modification callbacks that NSO invokes before (pre) or after (post) create. These callbacks work outside of the scope tracked by FASTMAP. That is, changes done in pre- and post-modification do not automatically get removed during update or delete of the service instance.

For example, you can use the pre-modification callback to check the service prerequisites (pre-check) or make changes that you want persisted even after the service is removed, such as enabling some global device feature. The latter may be required when NSO is not the only system managing the device and removing the feature configuration would break non-NSO managed services.

Similarly, you might use post-modification to reset the configuration to some default after the service is removed. Say the service configures an interface on a router for customer VPN. However, when the service is deprovisioned (removed), you don't want to simply erase the interface configuration. Instead, you want to put it in shutdown and configure it for a special, unused VLAN. The post-modification callback allows you to achieve this goal.

The main difference from create callback is that pre- and post-modification are called on update and delete, as well as service create. Since the service data node may no longer exist in case of delete, the API for these callbacks does not supply the service object. Instead, the callback receives the operation and key path to the service instance. See the following API signatures for details.

    @Service.pre_modification
    def cb_pre_modification(self, tctx, op, kp, root, proplist): ...

    @Service.create
    def cb_create(self, tctx, root, service, proplist): ...

    @Service.post_modification
    def cb_post_modification(self, tctx, op, kp, root, proplist): ...

The Python callbacks use the following function arguments:

    @ServiceCallback(servicePoint = "...",
                     callType = ServiceCBType.PRE_MODIFICATION)
    public Properties preModification(ServiceContext context,
                                      ServiceOperationType operation,
                                      ConfPath path,
                                      Properties opaque)
                                      throws DpCallbackException;

    @ServiceCallback(servicePoint="...",
                     callType=ServiceCBType.CREATE)
    public Properties create(ServiceContext context,
                             NavuNode service,
                             NavuNode ncsRoot,
                             Properties opaque)
                             throws DpCallbackException;

    @ServiceCallback(servicePoint = "...",
                     callType = ServiceCBType.POST_MODIFICATION)
    public Properties postModification(ServiceContext context,
                                       ServiceOperationType operation,
                                       ConfPath path,
                                       Properties opaque)
                                       throws DpCallbackException;

The Java callbacks use the following function arguments:

See examples.ncs/development-guide/services/post-modification-py and examples.ncs/development-guide/services/post-modification-java examples for a sample implementation of the post-modification callback.

Additionally, you may implement these callbacks with templates. Refer to the section called “Service callpoints and templates” for details.

FASTMAP greatly simplifies service code, so it usually only needs to deal with the initial mapping. NSO achieves this by first discarding all the configuration performed during the create callback of the previous run. In other words, the service create code always starts anew, with a blank slate.

If you need to keep some private service data across runs of the create callback, or pass data between callbacks, such as pre- and post-modification, you can use opaque properties.

The opaque object is available in the service callbacks as an argument, typically named proplist (Python) or opaque (Java). It contains a set of named properties with their corresponding values.

If you wish to use the opaque properties, it is crucial that your code returns the properties object from the create call, otherwise, the service machinery will not save the new version.

Compared to pre- and post-modification callbacks, which also persist data outside of FASTMAP, NSO deletes the opaque data when the service instance is deleted, unlike with the pre- and post-modification data.

    @Service.create
    def cb_create(self, tctx, root, service, proplist):
        intf = None
        # proplist is of type list[tuple[str, str]]
        for pname, pvalue in proplist:
            if pname == 'INTERFACE':
                intf = pvalue

        if intf is None:
            intf = '...'
            proplist.append('INTERFACE', intf)

        return proplist

    public Properties create(ServiceContext context,
                             NavuNode service,
                             NavuNode ncsRoot,
                             Properties opaque)
                             throws DpCallbackException {
        // In Java API, opaque is null when service instance is first created.
        if (opaque == null) {
            opaque = new Properties();
        }
        String intf = opaque.getProperty("INTERFACE");
        if (intf == null) {
            intf = "...";
            opaque.setProperty("INTERFACE", intf);
        }

        return opaque;
    }

The examples.ncs/development-guide/services/post-modification-py and examples.ncs/development-guide/services/post-modification-java examples showcase the use of opaque properties.

NSO by default enables concurrent scheduling and execution of services to maximize throughput. However, concurrent execution can be problematic for non-thread-safe services or services that are known to always conflict with themselves or other services, such as when they read and write the same shared data. See NSO Concurrency Model for details.

To prevent NSO from scheduling a service instance together with an instance of another service, declare a static conflict in the service model, using the ncs:conflicts-with extension. The following example shows a service with two declared static conflicts, one with itself and one with another service, named other-service.

        list example-service {
          key name;
          leaf name {
            type string;
          }
          uses ncs:service-data;
          ncs:servicepoint example-service {
            ncs:conflicts-with example-service;
            ncs:conflicts-with other-service;
          }
        }

This means each service instance will wait for other service instances that have started sooner than this one (and are of example-service or other-service type) to finish before proceeding.

FASTMAP knows that a particular piece of configuration belongs to a service instance, allowing NSO to revert the change as needed. But what happens when several service instances share a resource that may or may not exist before the first service instance is created? If the service implementation naively checks for existence and creates the resource when it is missing, then the resource will be tracked with the first service instance only. If, later on, this first instance is removed, then the shared resource is also removed, affecting all other instances.

A well-known solution to this kind of problem is reference counting. NSO uses reference counting by default with the XML templates and Python Maagic API, while in Java Maapi and Navu APIs, the sharedCreate(), sharedSet(), and sharedSetValues() functions need to be used.

When enabled, the reference counter allows FASTMAP algorithm to keep track of the usage and only delete data when the last service instance referring to this data is removed. Furthermore, containers and list items created using the sharedCreate() and sharedSetValues() functions also get an additional attribute called backpointer. (But this functionality is currently not available for individual leafs.)

Backpointer points back to the service instance that created the entity in the first place. This makes it possible to look at part of the configuration, say under /devices tree, and answer the question: which parts of the device configuration were created by which service?

To see reference counting in action, start the examples.ncs/implement-a-service/iface-v3 example with make demo and configure a service instance.

admin@ncs(config)# iface instance1 device c1 interface 0/1 ip-address 10.1.2.3 cidr-netmask 28
admin@ncs(config)# commit

Then configure another service instance with the same parameters and use the display service-meta-data pipe to show the reference counts and backpointers:

admin@ncs(config)# iface instance2 device c1 interface 0/1 ip-address 10.1.2.3 cidr-netmask 28
admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance2 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
    }
}
admin@ncs(config)# commit and-quit
admin@ncs# show running-config devices device c1 config interface\
 GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ip address 10.1.2.3 255.255.255.240
   ! Refcount: 2
   ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
   ip dhcp snooping trust
  exit
 !
!

Notice how commit dry-run produces no new device configuration but the system still tracks the changes. If you wish, remove the first instance and verify the GigabitEthernet 0/1 configuration is still there, but is gone when you also remove the second one.

But what happens if the two services produce different configuration for the same node? Say, one sets the IP address to 10.1.2.3 and the other to 10.1.2.4. Conceptually, these two services are incompatible and instantiating both at the same time produces broken configuration (instantiating the second service instance breaks the configuration for the first). What is worse is that the current configuration depends on the order the services were deployed or re-deployed. For example, re-deploying the first service will change the configuration from 10.1.2.4 back to 10.1.2.3 and vice versa. Such inconsistencies break the declarative configuration model and really should be avoided.

In practice, however, NSO does not prevent services from producing such configuration. But note that we strongly recommend against it and that there are associated limitations, such as service un-deploy not reverting configuration to that produced by the other instance (but when all services are removed, the original configuration is still restored). The commit | debug service pipe command warns about any such conflict that it finds but may miss conflicts on individual leafs. The best practice is to use integration tests in the service development life cycle to ensure there are no conflicts, especially when multiple teams develop their own set of services that are to be deployed on the same NSO instance.

Much like a service in NSO can provisioning device configurations, it can also provision other, non-device data, as well as other services. We call the approach of services provisioning other services service stacking and the services that are involved — stacked.

Service stacking concepts usually come into play for bigger, more complex services. There are a number of reasons why you might prefer stacked services to a single monolithic one:

Stacked services are also the basis for LSA, which takes this concept even further. See Layered Service Architecture for details.

The standard naming convention with stacked services distinguishes between a Resource-Facing Service (RFS), that directly configures one or more devices, and a Customer-Facing Service (CFS), that is the top-level service, configuring only other services, not devices. There can be more than two layers of services in the stack, too.

While NSO does not prevent a single service from configuring devices as well as services, in the majority of cases this results in a less clean design and is best avoided.

Overall, creating stacked services is very similar to the non-stacked approach. First, you can design the RFS services as usual. Actually, you might take existing services and reuse those. These then become your lower-level services, since they are lower in the stack.

Then you create a higher-level service, say a CFS, that configures another service, or a few, instead of a device. You can even use a template-only service to do that, such as:

<config-template xmlns="http://tail-f.com/ns/config/1.0"
                 servicepoint="top-level-service">
  <iface xmlns="http://com/example/iface">
    <name>instance1</name>
    <device>c1</device>
    <interface>0/1</interface>
    <ip-address>10.1.2.3</ip-address>
    <cidr-netmask>28</cidr-netmask>
  </iface>
</config>

The preceding example references an existing iface service, such as the one in the examples.ncs/implement-a-service/iface-v3 example. The output shows hard-coded values but you can change those as you would for any other service.

In practice, you might find it beneficial to modularize your data model and potentially reuse parts in both, the lower- and higher-level service. This avoids duplication while still allowing you to directly expose some of the lower-level service functionality through the higher-level model.

The most important principle to keep in mind is that the data created by any service is owned by that service, regardless of how the mapping is done (through code or templates). If the user deletes a service instance, FASTMAP will automatically delete whatever the service created, including any other services. Likewise, if the operator directly manipulates service data that is created by another service, the higher-level service becomes out of sync. The check-sync service action checks this for services as well as devices.

In stacked service design, the lower-level service data is under the control of the higher-level service and must not be directly manipulated. Only the higher-level service may manipulate that data. However, two higher-level services may manipulate the same structures, since NSO performs reference counting (see the section called “Reference Counting Overlapping Configuration”).

This section lists some specific advice for implementing services, as well as any known limitations you might run into.

You may also obtain some useful information by using the debug service commit pipe command, such as commit dry-run | debug service. The command display the net effect of the service create code, as well as issue warnings about potentially problematic usage of overlapping shared data.

A very common situation, when NSO is deployed in an existing network, is that the network already has services implemented. These services may have been deployed manually or through an older provisioning system. To take full advantage of the new system, you should consider importing the existing services into NSO. The goal is to use NSO to manage existing service instances, along with adding new ones in the future.

The process of identifying services and importing them into NSO is called Service Discovery and can be broken down into the following high-level parts:

Ultimately, the problem that service discovery addresses is one of referencing or linking configuration to services. Since the network already contains target configuration, a new service instance in NSO produces no changes in the network. This means the new service in NSO by default does not own the network configuration. One side effect is that removing a service will not remove the corresponding device configuration, which is likely to interfere with service modification as well.

Figure 20. Service reconciliation

Some of the steps in the process can be automated, while others are mostly manual. The amount of work differs a lot depending on how structured and consistent the original deployment is.

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1
devices device c1
 config
  interface GigabitEthernet0/1
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance1 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
    }
}

admin@ncs# show running-config devices device c1 config interface\
 GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

import ncs
from openpyxl import load_workbook

def main()
    wb = load_workbook('services.xslx')
    sheet = wb[wb.sheetnames[0]]

    with ncs.maapi.single_write_trans('admin', 'python') as t:
        root = ncs.maagic.get_root(t)
        for sr in sheet.rows:
            # Suppose columns in spreadsheet are:
            # instance (A), device (B), interface (C), IP (D), mask (E)
            name = sr[0].value
            service = root.iface.create(name)
            service.device = sr[1].value
            service.interface = sr[2].value
            service.ip_address = sr[3].value
            service.cidr_netmask = sr[4].value

        t.apply()

main()

admin@ncs# show running-config iface | display xml
<config xmlns="http://tail-f.com/ns/config/1.0">
  <iface xmlns="http://com/example/iface">
    <name>instance1</name>
    <device>c1</device>
    <interface>0/1</interface>
    <ip-address>10.1.2.3</ip-address>
    <cidr-netmask>28</cidr-netmask>
  </iface>
</config>

devconfig = root.devices.device[service.device].config
intf = devconfig.interface.GigabitEthernet[service.interface]
netmask = intf.ip.address.primary.mask
cidr = IPv4Network(f'0.0.0.0/{netmask}').prefixlen

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

admin@ncs# iface instance1 re-deploy reconcile
admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 1
  ! Backpointer: [ /iface:iface[iface:name='instance1'] ]
  interface GigabitEthernet0/1
   ! Refcount: 1
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 3
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 3
   ! Originalvalue: 10.1.2.3
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

admin@ncs# show running-config devices device c1 config\
 interface GigabitEthernet 0/1 | display service-meta-data
devices device c1
 config
  ! Refcount: 2
  ! Backpointer: [ /iface:iface[iface:name='instance1'] /iface:iface[iface:name='instance2'] ]
  interface GigabitEthernet0/1
   ! Refcount: 2
   ip address 10.1.2.3 255.255.255.240
  exit
 !
!

admin@ncs(config)# no iface instance1
admin@ncs(config)# commit dry-run outformat native
native {
}
admin@ncs(config)# no iface instance2
admin@ncs(config)# commit dry-run outformat native
native {
    device {
        name c1
        data no interface GigabitEthernet0/1
    }
}

admin@ncs(config)# commit dry-run
cli {
    local-node {
        data +iface instance1 {
             +    device c1;
             +    interface 0/1;
             +    ip-address 10.1.2.3;
             +    cidr-netmask 28;
             +}
             +iface instance2 {
             +    device c1;
             +    interface 0/2;
             +    ip-address 10.2.2.3;
             +    cidr-netmask 28;
             +}
    }
}
admin@ncs(config)# commit

admin@ncs# iface instance1 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
}

admin@ncs# iface instance1 re-deploy reconcile

admin@ncs# iface instance2 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
    local-node {
        data  devices {
                   device c1 {
                       config {
                           interface {
                               GigabitEthernet 0/2 {
                                   ip {
                                       dhcp {
                                           snooping {
              -                                trust;
                                           }
                                       }
                                   }
                               }
                           }
                       }
                   }
               }

    }
}

iface instance2
 device       c1
 interface    0/2
 ip-address   10.2.2.3
 cidr-netmask 28
 variant      v3
!

admin@ncs# iface instance2 re-deploy reconcile\
 { discard-non-service-config } dry-run
cli {
}
admin@ncs# iface instance2 re-deploy reconcile

In some cases a service may need to rely on the actual device configurations to compute the changeset. It is often a requirement to pull the current device configurations from the network before executing such service. Doing a full sync-from on a number of devices is an expensive task, especially if it needs to be performed often. The alternative way in this case is using partial-sync-from.

In cases where multitude of service instances touch a device that is not entirely orchestrated using NSO, i.e. relying on the partial-sync-from feature described above, and the device needs to be replaced then all services need to be re-deployed. This can be expensive depending on the number of service instances. Partial-sync-to enables replacement of devices in a more efficient fashion.

Partial-sync-from and partial-sync-to actions allow to specify certain portions of the device's configuration to be pulled or pushed from or to the network, respectively, rather than the full config. These are more efficient operations on NETCONF devices and NEDs that support partial-show feature. NEDs that do not support partial-show feature will fall back to pulling or pushing the whole configuration.

Even though partial-sync-from and partial-sync-to allows to pull or push only a part of device's configuration, the actions are not allowed to break consistency of configuration in CDB or on the device as defined by the YANG model. Hence, extra consideration needs to be given to dependencies inside the device model. If some configuration item A depends on configuration item B in the device's configuration, pulling only A may fail due to unsatisfied dependency on B. In this case both A and B need to be pulled, even if the service is only interested in the value of A.

It is important to note that partial-sync-from and partial-sync-to clear the transaction ID of the device in NSO unless the whole configuration has been selected (e.g. /ncs:devices/ncs:device[ncs:name='ex0']/ncs:config). This ensures NSO does not miss any changes to other parts of the device configuration but it does make the device out of sync.

        ConfXMLParam[] params = new ConfXMLParam[] {
        new ConfXMLParamValue("ncs", "path", new ConfList(new ConfValue[] {
        new ConfBuf("/ncs:devices/ncs:device[ncs:name='ex0']/"
        + "ncs:config/r:sys/r:interfaces/r:interface[r:name='eth0']"),
        new ConfBuf("/ncs:devices/ncs:device[ncs:name='ex1']/"
        + "ncs:config/r:sys/r:dns/r:server")
        })),
        new ConfXMLParamLeaf("ncs", "suppress-positive-result")};
        ConfXMLParam[] result =
        maapi.requestAction(params, "/ncs:devices/ncs:partial-sync-from");