Nano Services for Staged Provisioning

By default, the plan outline consists of a single component, the self component, with the two states init and ready. It can be used to track the progress of the service as a whole. You can add any number of additional components and states to form the nano service.

The following YANG snippet, also part of the examples.ncs/development-guide/nano-services/basic-vrouter example, shows a plan outline with the two VM-provisioning states presented above:

module vrouter {
  prefix vr;

  identity vm-requested {
    base ncs:plan-state;
  }

  identity vm-configured {
    base ncs:plan-state;
  }

  identity vrouter {
    base ncs:plan-component-type;
  }

  ncs:plan-outline vrouter-plan {
    description "Plan for configuring a VM-based router";

    ncs:component-type "vr:vrouter" {
      ncs:state "vr:vm-requested";
      ncs:state "vr:vm-configured";
    }
  }
}

The first part contains a definition of states as identities, deriving from the ncs:plan-state base. These identities are then used with the ncs:plan-outline, inside an ncs:component-type statement. It is customary to use past tense for state names, for example “configured-vm” or “vm-configured” instead of “configure-vm” and “configuring-vm.”

At present, the plan contains one component and two states but no logic. If you wish to do any provisioning for a state, the state must declare a special nano create callback, otherwise, it just acts as a checkpoint. The nano create callback is similar to an ordinary create service callback, allowing service code or templates to perform configuration. To add a callback for a state, extend the definition in the plan outline:

ncs:state "vr:vm-requested" {
  ncs:create {
    ncs:nano-callback;
  }
}

The service automatically enters each state one by one when a new service instance is configured. However, for the “vm-configured” state, the service should wait until the router VM has had the time to boot and is ready to accept a new configuration. An ncs:pre-condition statement in YANG provides this functionality. Until the condition becomes fulfilled, the service will not advance to that state.

The following YANG code instructs the nano service to check the value of the vm-up-and-running leaf, before entering and performing the configuration for a state.

ncs:state "vr:vm-configured" {
  ncs:create {
    ncs:nano-callback;
    ncs:pre-condition {
      ncs:monitor "$SERVICE" {
        ncs:trigger-expr "vm-up-and-running = 'true'";
      }
    }
  }
}

The main reason for defining multiple nano service states is to specify what part of the overall configuration belongs in each state. For the VM-router example, that entails splitting the configuration into a part for deploying a VM on a virtual infrastructure and a part for configuring it. In this case, a router VM is requested simply by adding an entry to a list of VM requests, while making the API calls is left to an external component, such as the VNF Manager.

If a state defines a nano callback, you can register a configuration template to it. The XML template file is very similar to an ordinary service template but requires the additional componenttype and state attributes in the config-template root element. These attributes identify which component and state in the plan outline the template belongs to, for example:

<config-template xmlns="http://tail-f.com/ns/config/1.0"
                 servicepoint="vrouter-servicepoint"
                 componenttype="vr:vrouter"
                 state="vr:vm-configured">
  <devices xmlns="http://tail-f.com/ns/ncs">
    <!-- ... -->
  </devices>
</config-template>

Likewise, you can implement a callback in the service code. The registration requires you to specify the component and state, as the following Python example demonstrates:

class NanoApp(ncs.application.Application):
    def setup(self):
        self.register_nano_service('vrouter-servicepoint',  # Service point
                                   'vr:vrouter',            # Component
                                   'vr:vm-requested',       # State
                                   NanoServiceCallbacks)

The selected NanoServiceCallbacks class then receives callbacks in the cb_nano_create() function:

class NanoServiceCallbacks(ncs.application.NanoService):
    @ncs.application.NanoService.create
    def cb_nano_create(self, tctx, root, service, plan, component, state,
                       proplist, component_proplist):
        ...

The component and state parameters allow the function to distinguish calls for different callbacks when registered for more than one.

For most flexibility, each state defines a separate callback, allowing you to implement some with a template and others with code, all as part of the same service. You may even use Java instead of Python, as explained in the section called “Nano Service Callbacks”.

The set of states used in the plan outline describe the stages that a service instance goes through during provisioning. Naturally, these are service-specific, which presents a problem if you just want to tell whether a service instance is still provisioning or has already finished. It requires the knowledge of which state is the last, final one, making it hard to check in a generic way.

That is why each service component must have the built-in ncs:init state as the first state and ncs:ready as the last state. Using the two built-in states allows for interoperability with other services and tools. The following is a complete four-state plan outline for the VM-based router service, with the two states added:

ncs:plan-outline vrouter-plan {
  description "Plan for configuring a VM-based router";

  ncs:component-type "vr:vrouter" {
    ncs:state "ncs:init";
    ncs:state "vr:vm-requested" {
      ncs:create {
        ncs:nano-callback;
      }
    }
    ncs:state "vr:vm-configured" {
      ncs:create {
        ncs:nano-callback;
        ncs:pre-condition {
          ncs:monitor "$SERVICE" {
            ncs:trigger-expr "vm-up-and-running = 'true'";
          }
        }
      }
    }
    ncs:state "ncs:ready";
  }
}

For the service to use it, the plan outline must be linked to a service point with the help of a behavior tree. The main purpose of a behavior tree is to allow a service to dynamically instantiate components, based on service parameters. Dynamic instantiation is not always required and the behavior tree for a basic, static, single-component scenario boils down to the following:

ncs:service-behavior-tree vrouter-servicepoint {
  description "A static, single component behavior tree";
  ncs:plan-outline-ref "vr:vrouter-plan";
  ncs:selector {
    ncs:create-component "'vrouter'" {
      ncs:component-type-ref "vr:vrouter";
    }
  }
}

This behavior tree always creates a single “vrouter” component for the service. The service point is provided as an argument to the ncs:service-behavior-tree statement, while the ncs:plan-outline-ref statement provides the name for the plan outline to use.

The following figure visualizes the resulting service plan and its states.

Figure 61. Virtual router provisioning plan

Along with the behavior tree, a nano service also relies on the ncs:nano-plan-data grouping in its service model. It is responsible for storing state and other provisioning details for each service instance. Other than that, the nano service model follows the standard YANG definition of a service:

list vrouter {
  description "Trivial VM-based router nano service";

  uses ncs:nano-plan-data;
  uses ncs:service-data;
  ncs:servicepoint vrouter-servicepoint;

  key name;
  leaf name {
    type string;
  }

  leaf vm-up-and-running {
    type boolean;
    config false;
  }
}

This model includes the operational vm-up-and-running leaf, that the example plan outline depends on. In practice, however, a plan outline is more likely to reference values provided by another part of the system, such as the actual, externally provided, state of the provisioned VM.

A nano service does not directly use its service point for configuration. Instead, the service point invokes a behavior tree to generate a plan, and the service starts executing according to this plan. As it reaches a certain state, it performs the relevant configuration for that state.

For example, when you create a new instance of the VM-router service, the vm-up-and-running leaf is not set, so only the first part of the service runs. Inspecting the service instance plan reveals the following:

admin@ncs# show vrouter vr-01 plan
                                                                                     POST
                  BACK                                                               ACTION
TYPE     NAME     TRACK  GOAL  STATE          STATUS       WHEN                 ref  STATUS
---------------------------------------------------------------------------------------------
self     self     false  -     init           reached      2023-08-11T07:45:20  -    -
                               ready          not-reached  -                    -    -
vrouter  vrouter  false  -     init           reached      2023-08-11T07:45:20  -    -
                               vm-requested   reached      2023-08-11T07:45:20  -    -
                               vm-configured  not-reached  -                    -    -
                               ready          not-reached  -                    -    -

Since neither the “init” nor the “vm-requested” states have any pre-conditions, they are reached right away. In fact, NSO can optimize it into a single transaction (this behavior can be disabled if you use forced commits, discussed later on).

But the process has stopped at the “vm-configured” state, denoted by the not-reached status in the output. It is waiting for the pre-condition to become fulfilled with the help of a kicker. The job of the kicker is to watch the value and perform an action, the reactive re-deploy, when the conditions are satisfied. The kickers are managed by the nano service subsystem: when an unsatisfied precondition is encountered, a kicker is configured, and when the precondition becomes satisfied, the kicker is removed.

You may also verify, through the get-modifications action, that only the first part, the creation of the VM, was performed:

admin@ncs# vrouter vr-01 get-modifications
cli {
    local-node {
        data +vm-instance vr-01 {
              +    type csr-small;
              +}

    }
}

At the same time, a kicker was installed under the kickers container but you may need to use the unhide debug command to inspect it. More information on kickers in general is available in Kicker .

At a later point in time, the router VM becomes ready, and the vm-up-and-running leaf is set to a true value. The installed kicker notices the change and automatically calls the reactive-re-deploy action on the service instance. In turn, the service gets fully deployed.

admin@ncs# show vrouter vr-01 plan
                                                                                 POST
                  BACK                                                           ACTION
TYPE     NAME     TRACK  GOAL  STATE          STATUS   WHEN                 ref  STATUS
-----------------------------------------------------------------------------------------
self     self     false  -     init           reached  2023-08-11T07:45:20  -    -
                               ready          reached  2023-08-11T07:47:36  -    -
vrouter  vrouter  false  -     init           reached  2023-08-11T07:45:20  -    -
                               vm-requested   reached  2023-08-11T07:45:20  -    -
                               vm-configured  reached  2023-08-11T07:47:36  -    -
                               ready          reached  2023-08-11T07:47:36  -    -

The get-modifications output confirms this fact. It contains the additional IP address configuration, performed as part of the “vm-configured” step:

admin@ncs# vrouter vr-01 get-modifications
cli {
    local-node {
        data +vm-instance vr-01 {
             +    type    csr-small;
             +    address 198.51.100.1;
             +}
    }
}

The “ready” state has no additional pre-conditions, allowing NSO to reach it along with the “vm-configured” state. This effectively breaks the provisioning process into two steps. To break it down further, simply add more states with corresponding pre-conditions and create logic.

Other than staged provisioning, nano services act the same as other services, allowing you to use the service check-sync and similar actions, for example. But please note the un-deploy and re-deploy actions may behave differently than expected, as they deal with provisioning. Chiefly, a re-deploy reevaluates the pre-conditions, possibly generating a different configuration if a pre-condition depends on operational values that have changed. The un-deploy action, on the other hand, removes all of the recorded modifications, along with the generated plan.

When a service's plan component changes state, the plan-state-change notification is generated with the new state of the plan. It includes the status, which indicates one of not-reached, reached, or failed. The notification is sent when the state is created, modified, or deleted, depending on the configuration. For reference on the structure and all the fields present in the notification, please see the YANG model in the tailf-ncs-plan.yang file.

As a common use-case, an event with status reached for the self component ready state signifies that all nano service components have reached their ready state and provisioning is complete. A simple example of this scenario is included in the examples.ncs/development-guide/nano-services/netsim-vrouter/demo.py Python script, using RESTCONF.

To enable the plan-state-change notifications to be sent, you must enable them for a specific service in NSO. For example, can load the following configuration into the CDB as an XML initialization file:

<services xmlns="http://tail-f.com/ns/ncs">
  <plan-notifications>
    <subscription>
      <name>nano1</name>
      <service-type>/vr:vrouter</service-type>
      <component-type>self</component-type>
      <state>ready</state>
      <operation>modified</operation>
    </subscription>
    <subscription>
      <name>nano2</name>
      <service-type>/vr:vrouter</service-type>
      <component-type>self</component-type>
      <state>ready</state>
      <operation>created</operation>
    </subscription>
  </plan-notifications>
</services>

This configuration enables notifications for the self component's ready state when created or modified.

When a service is committed through the commit queue, this notification acts as a reference regarding the state of the service. Notifications are sent when the service commit queue item is waiting to run, executing, waiting to be unlocked, completed, failed, or deleted. More details on the service-commit-queue-event notification content can be found in the YANG model inside tailf-ncs-services.yang .

For example, the failed event can be used to detect that a nano service instance deployment failed because a configuration change committed through the commit queue has failed. Measures to resolve the issue can then be taken and the nano service instance can be re-deployed. A simple example of this scenario is included in the examples.ncs/development-guide/nano-services/netsim-vrouter/demo.py Python script where the service is committed through the commit queue, using RESTCONF. By design, the configuration commit to a device fails, resulting in a commit-queue-notification with the failed event status for the commit queue item.

To enable the service-commit-queue-event notifications to be sent, you can load the following example configuration into NSO, as an XML initialization file or some other way:

<services xmlns="http://tail-f.com/ns/ncs">
  <commit-queue-notifications>
    <subscription>
      <name>nano1</name>
      <service-type>/vr:vrouter</service-type>
    </subscription>
  </commit-queue-notifications>
</services>

The following examples demonstrate the usage and sample events for the notification functionality, described in this section, using RESTCONF, NETCONF, and CLI northbound interfaces.

RESTCONF subscription request using curl:

$ curl -isu admin:admin -X GET -H "Accept: text/event-stream"
    http://localhost:8080/restconf/streams/service-state-changes/json

data: {
data:   "ietf-restconf:notification": {
data:     "eventTime": "2021-11-16T20:36:06.324322+00:00",
data:     "tailf-ncs:service-commit-queue-event": {
data:       "service": "/vrouter:vrouter[name='vr7']",
data:       "id": 1637135519125,
data:       "status": "completed",
data:       "trace-id": "vr7-1"
data:     }
data:   }
data: }

data: {
data:   "ietf-restconf:notification": {
data:     "eventTime": "2021-11-16T20:36:06.728911+00:00",
data:     "tailf-ncs:plan-state-change": {
data:       "service": "/vrouter:vrouter[name='vr7']",
data:       "component": "self",
data:       "state": "tailf-ncs:ready",
data:       "operation": "modified",
data:       "status": "reached",
data:       "trace-id": "vr7-1"
data:     }
data:   }
data: }

See the section called “Streams” in Northbound APIs for further reference.

NETCONF create subscription using netconf-console:

$ netconf-console create-subscription=service-state-changes

<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
  <eventTime>2021-11-16T20:36:06.324322+00:00</eventTime>
  <service-commit-queue-event xmlns="http://tail-f.com/ns/ncs">
    <service xmlns:vr="http://com/example/vrouter">/vr:vrouter[vr:name='vr7']</service>
    <id>1637135519125</id>
    <status>completed</status>
    <trace-id>vr7-1</trace-id>
  </service-commit-queue-event>
</notification>
<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
  <eventTime>2021-11-16T20:36:06.728911+00:00</eventTime>
  <plan-state-change xmlns="http://tail-f.com/ns/ncs">
    <service xmlns:vr="http://com/example/vrouter">/vr:vrouter[vr:name='vr7']</service>
    <component>self</component>
    <state>ready</state>
    <operation>modified</operation>
    <status>reached</status>
    <trace-id>vr7-1</trace-id>
  </plan-state-change>
</notification>

See the section called “Notification Capability” in Northbound APIs for further reference.

CLI show received notifications using ncs_cli:

$ ncs_cli -u admin -C <<<'show notification stream service-state-changes'

notification
 eventTime 2021-11-16T20:36:06.324322+00:00
 service-commit-queue-event
  service /vrouter[name='vr17']
  id 1637135519125
  status completed
  trace-id vr7-1
 !
!
notification
 eventTime 2021-11-16T20:36:06.728911+00:00
 plan-state-change
  service /vrouter[name='vr7']
  component self
  state ready
  operation modified
  status reached
  trace-id vr7-1
 !
!

You have likely noticed the trace-id field at the end of the example notifications above. The trace id is an optional but very useful parameter when committing the service configuration. It helps you trace the commit in the emitted log messages and the service-state-changes stream notifications. The above notifications, taken from the examples.ncs/development-guide/nano-services/netsim-vrouter example, are emitted after applying a RESTCONF plain patch:

$ curl -isu admin:admin -X PATCH
  -H "Content-type: application/yang-data+json"
  'http://localhost:8080/restconf/data?commit-queue=sync&trace-id=vr7-1'
  -d '{ "vrouter:vrouter": [ { "name": "vr7" } ] }'
          

Note that the trace id is specified as part of the URL. If missing, NSO will generate and assign one on its own.

Adding new components to the behavior tree will create the new components during the next re-deploy (synthetization) and execute the states in the new components as is normally done.

When removing components from the behavior tree, the components that are removed are set to backtracking and are backtracked fully before they are removed from the plan.

When you remove a component, do so carefully so that any callbacks, post actions or any other model data that the component depends on are not removed until all instances of the old component are removed.

If the identity for a component type is removed, then NSO removes the component from the database when upgrading the package. If this happens, the component is not backtracked and the reverse diffsets are not applied.

Replacing components in the behavior tree is the same as having unrelated components that are deleted and added in the same update. The deleted components are backtracked as far as possible, and then the added components are created and their states executed in order.

In some cases, this is not the desired behaviour when replacing a component. For example, if you only want to rename a component, backtracking and then adding the component again might make NSO push unnecessary changes to the network or run delete callbacks and post actions that should not be run. To remedy this, you might add the ncs:deprecates-component statements to the new component, detailing which components it replaces. NSO then skips the backtracking of the old component and just applies all reverse diffsets of the deprecated component. In the same re-deploy, it then executes the new component as usual. Therefore, if the new component produces the same configuration as the old component, nothing is pushed to the network.

If any of the deprecated components are backtracking, the backtracking will be handled before the component is removed. When there are multiple components that are deprecated in the same update, the components will not be removed, as detailed above, until all of them are done backtracking (if any one of them are backtracking).

When adding or removing states in a component, the component is backtracked before a new component with the new states is added and executed. If the updated component produces the same configuration as the old one (and no preconditions halt the execution), this should lead to no configuration being pushed to the network. So, if changes to the states are done, you need to take care when writing the preconditions and post actions for a component if no new changes should be pushed to the network.

Any changes to the already present states that are kept in the updated component will not have their configuration updated until the new component is created, which happens after the old one has been fully backtracked.

For a component where only the configuration for one or more states have changed, the synthetization process will update the component with the new configuration and make sure that any new callbacks or similar are called during future execution of the component.

Back-tracking is completely automatic and occurs in the following scenarios:

For each RFM loop, NSO traverses each component and state in order. For each non-satisfied create pre-condition, a kicker is started that monitors and triggers when the pre-condition becomes satisfied.

While traversing the states, a create pre-condition that was previously satisfied may become un-satisfied. If there are subsequent reached states that contain reverse diff-sets, then the component must be set to back-tracking mode. The back-tracking mode has as its goal to revert all changes up to the state which originally failed to satisfy its create pre-condition. While back-tracking, the delete pre-condition for each state is evaluated, if it exists. If the delete pre-condition is satisfied, the state's reverse diff-set is applied, and the next state is considered. If the delete pre-condition is not satisfied, a kicker is created to monitor this delete pre-condition. When the kicker triggers, a reactive-re-deploy is called and the back-tracking will continue until the goal is reached.

When the back-tracking plan component has reached its goal state, the component is set to normal mode again. The state's create pre-condition is evaluated and if it is satisfied the state is entered or otherwise a kicker is created as described above.

In some circumstances a complete plan component is removed (for example, if the service input parameters are changed). If this happens, the plan component is checked if it contains reached states that contain reverse diff-sets.

If the removed component contains reached states with reverse diff-sets, the deletion of the component is deferred and the component is set to back-tracking mode.

In this case, there is no specified goal state for the back-tracking. This means that when all the states have been reverted, the component is automatically deleted.

If a service is deleted, all components are set to back-tracking mode. The service becomes a zombie, storing away its plan states so that the service configuration can be removed.

All components of a deleted service are set in backtracking mode.

When a component becomes completely back-tracked, it is removed.

When all components in the plan are deleted, the service is removed.

A nano service behavior tree is a data structure defined for each service type. Without a behavior tree defined for the service point, the nano service cannot execute. It is the behavior tree that defines the currently executing nano-plan with its components.

The purpose of a behavior tree is to have a declarative way to specify how the service's input parameters are mapped to a set of component instances.

A behavior tree is a directed tree in which the nodes are classified as control flow nodes and execution nodes. For each pair of connected nodes, the outgoing node is called parent and the incoming node is called child. A control flow node has zero or one parent and at least one child, and the execution nodes have one parent and no children.

There is exactly one special control flow node called the root, which is the only control flow node without a parent.

This definition implies that all interior nodes are control flow nodes, and all leaves are execution nodes. When creating, modifying, or deleting a nano service, NSO evaluates the behavior tree to render the current nano plan for the service. This process is called synthesizing the plan.

The control flow nodes have a different behavior, but in the end, they all synthesize its children in zero or more instances. When the a control flow node is synthesized, the system executes its rules for synthesizing the node's children. Synthesizing an execution node adds the corresponding plan component instance to the nano service's plan.

All control flow and execution nodes may define pre-conditions, which must be satisfied to synthesize the node. If a pre-condition is not satisfied, a kicker is started to monitor the pre-condition.

All control flow and execution nodes may define an observe monitor which results in a kicker being started for the monitor when the node is synthesized.

If an invocation of an RFM loop (for example, a re-deploy) synthesizes the behavior tree and a pre-condition for a child is no longer satisfied, the sub-tree with its plan-components is removed (that is, the plan-components are set to back-tracking mode).

The following control flow nodes are defined:

There is just one type of execution node:

It is recommended to keep the behavior tree as flat as possible. The most trivial case is when the behavior tree creates a static nano-plan, that is, all the plan-components are defined and never removed. The following is an example of such a behavior tree:

Having a selector on root implies that all plan-components are created if they don't have any pre-conditions, or for which the pre-conditions are satisfied.

An example of a more elaborated behavior tree is the following:

This behavior tree has a selector node as the root. It will always synthesize the "base-config" plan component and then evaluate then pre-condition for the selector child. If that pre-condition is satisfied, it then creates four other plan-components.

The multiplier control flow node is used when a plan component of a certain type should be cloned into several copies depending on some service input parameters. For this reason, the multiplier node defines a foreach, a when, and a variable. The foreach is evaluated and for each node in the nodeset that satisfies the when, the variable is evaluated as the outcome. The value is used for parameter substitution to a unique name for a duplicated plan component.

The value is also added to the nano service opaque which enables the individual state nano service create() callbacks to retrieve the value.

Variables might also have “when” expressions, which are used to decide if the variable should be added to the list of variables or not.

Pre-conditions are what drives the execution of a nano service. A pre-condition is a prerequisite for a state to be executed or a component to be synthesized. If the pre-condition is not satisfied, it is then turned into a kicker which in turn re-deploys the nano service once the condition is fulfilled.

When working with pre-conditions, you need to be aware that they work a bit differently when used as a kicker to re-deploy the service and when they are used in the execution of the service. When the pre-condition is used in the re-deploy kicker, it then works as explained in the kicker documentation (that is, the trigger expression is evaluated before and after the change-set of the commit when the monitored nodeset is changed). When used during the execution of a nano service, you can only evaluate it on the current state of the database, which means that it only checks that the monitor returns a nodeset of one or more nodes and that trigger expression (if there is one) is fulfilled for any of the nodes in the nodeset.

Support for pre-conditions checking, if a node has been deleted, is handled a bit differently due to the difference in how the pre-condition is evaluated. Kickers always trigger for changed nodes (add, deleted, or modified) and can check that the node was deleted in the commit that triggered the kicker. While in the nano service evaluation, you only have the current state of the database and the monitor expression will not return any nodes for evaluation of the trigger expression, consequently evaluating the pre-condition to false. To support deletes in both cases, you can create a pre-condition with a monitor expression and a child node ncs:trigger-on-delete which then both create a kicker that checks for deletion of the monitored node and also does the right thing in the nano service evaluation of the pre-condition. For example, you could have the following component:

            ncs:component "base-config" {
              ncs:state "init" {
                ncs:delete {
                  ncs:pre-condition {
                    ncs:monitor "/devices/device[name='test']" {
                      ncs:trigger-on-delete;
                    }
                  }
                }
              }
              ncs:state "ready";
            }

The component would only trigger the init states delete pre-condition when the device named test is deleted.

It is possible to add multiple monitors to a pre-condition by using the ncs:all or ncs:any extensions. Both extensions take one or multiple monitors as argument. A pre-condition using the ncs:all extension is satisfied if all monitors given as arguments evaluate to true. A pre-condition using the ncs:any extension is satisfied if at least one of the monitors given as argument evaluates to true. The following component uses the ncs:all and ncs:any extensions for its self state's create and delete pre-condition, respectively:

          ncs:component "base-config" {
            ncs:state "init" {
              ncs:create {
                ncs:pre-condition {
                  ncs:all {
                    ncs:monitor $SERVICE/syslog {
                      ncs:trigger-expr: "current() = true"
                    }
                    ncs:monitor $SERVICE/dns {
                      ncs:trigger-expr: "current() = true"
                      }
                    }
                  }
                }
              }
              ncs:delete {
                ncs:pre-condition {
                  ncs:any {
                    ncs:monitor $SERVICE/syslog {
                      ncs:trigger-expr: "current() = false"
                    }
                    ncs:monitor $SERVICE/dns {
                      ncs:trigger-expr: "current() = false"
                      }
                    }
                  }
                }
              }
            }
            ncs:state "ready";
          }

The service opaque is a name-value list that can optionally be created/modified in some of the service callbacks, and then travels the chain of callbacks (pre-modification, create, post-modification). It is returned by the callbacks and stored persistently in the service private data. Hence, the next service invocation has access to the current opaque and can make subsequent read/write operations to the same object. The object is usually called opaque in Java and proplist in Python callbacks.

The nano services handle the opaque in a similar fashion, where a callback for every state has access to and can modify the opaque. However, the behavior tree can also define variables, which you can use in preconditions or to set component names. These variables are also available in the callbacks, as component properties. The mechanism is similar but separate from the opaque. While the opaque is a single service-instance-wide object set only from the service code, component variables are set in and scoped according to the behavior tree. That is, component properties contain only the behavior tree variables which are in scope when a component is synthesized.

For example, take the following behavior tree snippet:

    ncs:selector {
      ncs:variable "VAR1" {
        ncs:value-expr "'value1'";
      }
      ncs:create-component "'base-config'" {
        ncs:component-type-ref "t:base-config";
      }
      ncs:selector {
        ncs:variable "VAR2" {
          ncs:value-expr "'value2'";
        }
        ncs:create-component "'component1'" {
          ncs:component-type-ref "t:my-component";
        }
      }
    }

The callbacks for states in the “base-config” component only see the VAR1 variable, while those in “component1” see both VAR1 and VAR2 as component properties.

Additionally, both the service opaque and component variables (properties) are used to look up substitutions in nano service XML templates and in the behavior tree. If used in the behavior tree, the same rules apply for the opaque as for component variables. So, a value needs to contain single quotes if you wish to use it verbatim in preconditions and similar constructs, for example:

    proplist.append(('VARX', "'some value'"))

Using this scheme at an early state, such as the “base-config” component's “ncs:init”, you can have a callback that sets name-value pairs for all other states that are then implemented solely with templates and preconditions.

The nano service can have several callback registrations, one for each plan component state. But note that some states may have no callbacks at all. The state may simply act as a checkpoint, that some condition is satisfied, using pre-condition statements. A component's ncs:ready state is a good example of this.

The drawback with this flexible callback registration is that there must be a way for the NSO Service Manager to know if all expected nano service callbacks have been registered. For this reason, all nano service plan component states that require callbacks are marked with this information. When the plan is executed and the callback markings in the plan mismatch with the actual registrations, this results in an error.

All callback registrations in NSO require a daemon to be instantiated, such as a Python or Java process. For nano services, it is allowed to have many daemons where each daemon is responsible for a subset of the plan state callback registrations. The neat thing here is that it becomes possible to mix different callback types (Template/Python/Java) for different plan states.

The mixed callback feature caters to the case where most of the callbacks are templates and only some are Java or Python. This works well because nano services try to resolve the template parameters using the nano service opaque when applying a template. This is a unique functionality for nano services that makes Java or Python apply-template callbacks unnecessary.

You can implement nano service callbacks as Templates as well as Python, Java, Erlang, and C code. The following examples cover the implementation of Template, Python and Java.

A plan state template, if defined, replaces the need of a create() callback. In this case, there are no delete() callbacks and the status definitions must in this case be handled by the states delete pre-condition. The template must in addition to the servicepoint attribute have a component type and a state attribute to be registered on the plan state:

<config-template xmlns="http://tail-f.com/ns/config/1.0"
                 servicepoint="my-servicepoint"
                 componenttype="my:some-component"
                 state="my:some-state">
  <devices xmlns="http://tail-f.com/ns/ncs">
    <!-- ... -->
  </devices>
</config-template>

Specific to nano services, you can use parameters, such as $SOMEPARAM in the template. The system searches for the parameter value in the service opaque and in the component properties. If it is not defined, applying the template will fail.

A Python create() callback is very similar to its ordinary service counterpart. The difference is that it has additional arguments. plan refers to the synthesized plan, while component and state specify the component and state for which it is invoked. The proplist argument is the nano service opaque (same naming as for ordinary services) and component_proplist contains component variables, along with their values.

class NanoServiceCallbacks(ncs.application.NanoService):

    @ncs.application.NanoService.create
    def cb_nano_create(self, tctx, root, service, plan, component, state,
                       proplist, component_proplist):
        ...

    @ncs.application.NanoService.delete
    def cb_nano_delete(self, tctx, root, service, plan, component, state,
                       proplist, component_proplist):
        ...

In the majority of cases, you should not need to manage the status of nano states yourself. However, should you need to override the default behavior, you can set the status explicitly, in the callback, using code similar to the following :

        plan.component[component].state[state].status = 'failed'

The Python nano service callback needs a registration call for the specific servicepoint, componentType, and state that it should be invoked for.

class Main(ncs.application.Application):

    def setup(self):
        ...
        self.register_nano_service('my-servicepoint',
                                   'my:some-component',
                                   'my:some-state',
                                   NanoServiceCallbacks)

For Java, annotations are used to define the callbacks for the component states. The registration of these callbacks is performed by the ncs-java-vm. The NanoServiceContext argument contains methods for retrieving the component and state for the invoked callback as well as methods for setting the resulting plan state status.

public class myRFS {

    @NanoServiceCallback(servicePoint="my-servicepoint",
                         componentType="my:some-component",
                         state="my:some-state",
                         callType=NanoServiceCBType.CREATE)
    public Properties createSomeComponentSomeState(
                                    NanoServiceContext context,
                                    NavuNode service,
                                    NavuNode ncsRoot,
                                    Properties opaque,
                                    Properties componentProperties)
                                    throws DpCallbackException {
        // ...
    }

    @NanoServiceCallback(servicePoint="my-servicepoint",
                         componentType="my:some-component",
                         state="my:some-state",
                         callType=NanoServiceCBType.DELETE)
    public Properties deleteSomeComponentSomeState(
                                    NanoServiceContext context,
                                    NavuNode service,
                                    NavuNode ncsRoot,
                                    Properties opaque,
                                    Properties componentProperties)
                                    throws DpCallbackException {
        // ...
    }

Several componentType and state callbacks can be defined in the same Java class and are then registered by the same daemon.

In some scenarios, there is a need to be able to register a callback for a certain state in several components with different component types. For this reason, it is possible to register a callback with a wildcard, using “*” as the component type. The invoked state sends the actual component name to the callback, allowing the callback to still distinguish component types if required.

In Python, the component type is provided as an argument to the callback (component) and a generic callback is registered with an asterisk for a component, such as:

self.register_nano_service('my-servicepoint', '*', state, ServiceCallbacks)

In Java, you can perform the registration in the method annotation, as before. To retrieve the calling component type, use the NanoServiceContext.getComponent() method. For example:

    @NanoServiceCallback(servicePoint="my-servicepoint",
                         componentType="*", state="my:some-state",
                         callType=NanoServiceCBType.CREATE)
    public Properties genericNanoCreate(NanoServiceContext context,
                                        NavuNode service,
                                        NavuNode ncsRoot,
                                        Properties opaque,
                                        Properties componentProperties)
                                        throws DpCallbackException {

        String currentComponent = context.getComponent();
        // ...
    }

The generic callback can then act for the registered state in any component type.

The ordinary service pre/post modification callbacks still exist for nano services. They are registered as for an ordinary service and are invoked before the behavior tree synthetization, and after the last component/state invocation.

Registration of the ordinary create() will not fail for a nano service. But they will never be invoked.

When implementing a nano service, you might end up in a situation where a commit is needed between states in a component to make sure that something has happened before the service can continue executing. One example of such behaviour is if the service is dependent on the notifications from a device. In such a case, you can setup a notification kicker in the first state and then trigger a forced commit before any later states can proceed, therefore making sure that all future notifications are seen by the later states of the component.

To force a commit in between two states of a component, add the ncs:force-commit tag in a ncs:create or ncs:delete tag. See the following example:

              ncs:component "base-config" {
                ncs:state "init" {
                  ncs:create {
                    ncs:force-commit;
                  }
                }
                ncs:state "ready" {
                  ncs:delete {
                    ncs:force-commit;
                  }
                }
              }

When defining a nano service, it is assumed that the plan is stored under the service path, as ncs:plan-data is added to the service definition. When the service instance is deleted, the plan is moved to the zombie instead, since the instance has been removed and the plan cannot be stored under it anymore. When writing other services or when working with a nano service in general, you need to be aware that the plan for a service might be in one of these two places depending on if the service instance has been deleted or not.

To make it easier to work with a service, you can define a custom location for the plan and its history. In the ncs:service-behaviour-tree, you can specify that the plan should be stored outside of the service by setting the ncs:plan-location tag to a custom location. The location where the plan should be stored must be either a list or a container and include the ncs:plan-data tag. The plan data is then created in this location, no matter if the service instance has been deleted (turned into a zombie) or not, making it easy to base decisions on the state of the service as all plan queries can query the same plan.

You can use XPath with the ncs:plan-location statement. The XPath is evaluated based on the nano service context. When the list or container, which contains the plan, is nested under another list, the outer list instance must exist before creating the nano service. At the same time, the outer list instance of the plan location must also remain intact for further service's life-cycle management, such as redeployment, deletion etc. Otherwise, an error will be returned, logged, and any service interaction (create, re-deploy, delete, etc.) won't succeed.

  identity base-config {
    base ncs:plan-component-type;
  }

  list custom {
    description "Custom plan location example service.";

    key name;
    leaf name {
      tailf:info "Unique service id";
      tailf:cli-allow-range;
      type string;
    }

    uses ncs:service-data;
    ncs:servicepoint custom-plan-servicepoint;
  }

  list custom-plan {
    description "Custom plan location example plan.";

    key name;
    leaf name {
      tailf:info "Unique service id";
      tailf:cli-allow-range;
      type string;
    }

    uses ncs:nano-plan-data;
  }

  ncs:plan-outline custom-plan {
    description
      "Custom plan location example outline";

    ncs:component-type "p:base-config" {
      ncs:state "ncs:init";
      ncs:state "ncs:ready";
    }
  }

  ncs:service-behavior-tree custom-plan-location-servicepoint {
    description
      "Custom plan location example service behaviour three.";

    ncs:plan-outline-ref custom:custom-plan;
    ncs:plan-location "/custom-plan";

    ncs:selector {
      ncs:create-component "'base-config'" {
        ncs:component-type-ref "p:base-config";
      }
    }
  }

The commit queue feature, described in the section called “Commit Queue” in User Guide, allows for increased overall throughput of NSO by committing configuration changes into an outbound queue item instead of directly to affected devices. Nano services are aware of the commit queue and will make use of it, however, this interaction requires additional consideration.

When the commit queue is enabled and there are outstanding commit queue items, the network is lagging behind the CDB. The CDB is forward looking and shows the desired state of the network. Hence, the nano plan shows the desired state as well, since changes to reach this state may not have been pushed to the devices yet.

To keep the convergence of the nano service in sync with the commit queue, nano services behave more asynchronously:

The reason for such behavior is that commit queue items can fail. In case of a failure, the CDB and the network have diverged. In turn, the nano plan may have diverged and not reflect the actual network state if the failed commit queue item contained changes related to the nano service.

What is worse, the network may be left in an inconsistent state. To counter that, NSO supports multiple recovery options for the commit queue. Since NSO release 5.7, using the rollback-on-error is the recommended option, as it undoes all the changes that are part of the same transaction. If the transaction includes the initial service instance creation, the instance is removed as well. That is usually not desired for nano services. A nano service will avoid such removal by only committing the service intent (the instance configuration) in the initial transaction. In this case, the service avoids potential rollback, as it does not perform any device configuration in the same transaction but progresses solely through (reactive) re-deploy.

While error recovery helps keeping the network consistent, the end result remains that the requested change was not deployed. If a commit queue item with nano service related changes fails, that signifies a failure for the nano service and NSO does the following:

After such an event, manual intervention is required. If not using the rollback-on-error option or rollback transaction fails, please consult the section called “Commit Queue” in User Guide for the correct procedure to follow. Once the cause of the commit queue failure is resolved, you can manually resume the service progression by invoking the reactive-re-deploy action on a nano service or a zombie.