Managing Network Services

When NSO applies services to the network, NSO stores the service configuration along with resulting device configuration changes. This is used as a base for the FASTMAP algorithm which automatically can derive device configuration changes from a service change. So going back to the example L3 VPN above any part of volvo VPN instance can be modified. A simple change like changing the as-number on the service results in many changes in the network. NSO does this automatically.

ncs(config)# vpn l3vpn volvo as-number 65102
ncs(config-l3vpn-volvo)# commit dry-run outformat native
native {
    device {
        name ce0
        data no router bgp 65101
             router bgp 65102
              neighbor 192.168.1.2 remote-as 100
              neighbor 192.168.1.2 activate
              network 10.10.1.0
             !
...
ncs(config-l3vpn-volvo)# commit

Let us look at a more challenging modification. A common use-case is of course to add a new CE device and add that as an end-point to an existing VPN. Below follows the sequence to add two new CE devices and add them to the VPN's. (In the CLI snippets below we omit the prompt to enhance readability). First we add them to the topology.

top
!
topology connection c7
endpoint-1 device ce7 interface GigabitEthernet0/1 ip-address 192.168.1.25/30
endpoint-2 device pe3 interface GigabitEthernet0/0/0/2 ip-address 192.168.1.26/30
link-vlan 103
!
topology connection c8
endpoint-1 device ce8 interface GigabitEthernet0/1 ip-address 192.168.1.29/30
endpoint-2 device pe3 interface GigabitEthernet0/0/0/2 ip-address 192.168.1.30/30
link-vlan 104
!
ncs(config)#commit

Note well that the above just updates NSO local information on topological links. It has no effect on the network. The mapping for the L3 VPN services does a look-up in the topology connections to find the corresponding pe router.

Then we add them to the VPN's

top
!
vpn l3vpn ford
endpoint new-branch-office
ce-device    ce7
ce-interface GigabitEthernet0/5
ip-network   192.168.9.0/24
bandwidth    4500000
!
vpn l3vpn volvo
endpoint new-branch-office
ce-device    ce8
ce-interface GigabitEthernet0/5
ip-network   10.8.9.0/24
bandwidth    4500000
!

Before we send anything to the network, lets see look at the device configuration using dry-run. As you can see, both new CE devices are connected to the same PE router, but for different VPN customers.

ncs(config)#commit dry-run outformat native

And commit the configuration to the network

(config)#commit

Next we will show how NSO can be used to check if the service configuration in the network is up to date. In a new terminal window we connect directly to the device ce0 that is a Cisco device emulated by the tool ncs-netsim.

$ ncs-netsim cli-c ce0

We will now reconfigure an edge interface that we previously configured using NSO.

 enable
ce0# configure
Enter configuration commands, one per line. End with CNTL/Z.
ce0(config)# no policy-map volvo
ce0(config)# exit
ce0# exit

Going back to the terminal with NSO, check the status of the network configuration:

ncs# devices check-sync
sync-result {
    device ce0
    result out-of-sync
    info got: c5c75ee593246f41eaa9c496ce1051ea expected: c5288cc0b45662b4af88288d29be8667
...

ncs# vpn l3vpn * check-sync
vpn l3vpn ford check-sync
    in-sync true
vpn l3vpn volvo check-sync
    in-sync true

ncs# vpn l3vpn * deep-check-sync
vpn l3vpn ford deep-check-sync
    in-sync true
vpn l3vpn volvo deep-check-sync
    in-sync false

The CLI sequence above performs 3 different comparisons:

Notice that the service 'volvo' is out of sync from the service configuration. Use the check-sync outformat cli to see what the problem is:

ncs# vpn l3vpn volvo deep-check-sync outformat cli
cli  devices {
         devices {
             device ce0 {
                 config {
    +                ios:policy-map volvo {
    +                    class class-default {
    +                        shape {
    +                            average {
    +                                bit-rate 12000000;
    +                            }
    +                        }
    +                    }
    +                }
                 }
             }
         }
     }

Assume that a network engineer considers the real device configuration to be authoritative:

ncs# devices device ce0 sync-from
result true

And then restore the service:

ncs# vpn l3vpn volvo re-deploy dry-run { outformat native } 
native {
    device {
        name ce0
        data policy-map volvo
               class class-default
                shape average 12000000
               !
              !

    }
}
ncs# vpn l3vpn volvo re-deploy

In the same way as NSO can calculate any service configuration change it can also automatically delete the device configurations that resulted from creating services:

ncs(config)# no vpn l3vpn ford
ncs(config)# commit dry-run
cli  devices {
         device ce7
             config {
    -            ios:policy-map ford {
    -                class class-default {
    -                    shape {
    -                        average {
    -                            bit-rate 4500000;
    -                    }
    -                }
    -            }
    -        }
...

It is important to understand the two diffs shown above. The first diff as an output to show configuration shows the diff at service level. The second diff shows the output generated by NSO to clean up the device configurations.

Finally, we commit the changes to delete the service.

(config)# commit

Service instances live in the NSO data-store as well as a copy of the device configurations. NSO will maintain relationships between these two.

Show the configuration for a service

ncs(config)# show full-configuration vpn l3vpn
vpn l3vpn volvo
 as-number 65102
 endpoint branch-office1
  ce-device    ce1
  ce-interface GigabitEthernet0/11
  ip-network   10.7.7.0/24
  bandwidth    6000000
 !
...

You can ask NSO to list all devices that are touched by a service and vice versa:

ncs# show vpn l3vpn device-list
NAME   DEVICE LIST
----------------------------------------
volvo  [ ce0 ce1 ce4 ce8 pe0 pe2 pe3 ]


ncs# show devices device service-list
NAME  SERVICE LIST
-------------------------------------
ce0   [ "/l3vpn:vpn/l3vpn{volvo}" ]
ce1   [ "/l3vpn:vpn/l3vpn{volvo}" ]
ce2   [  ]
ce3   [  ]
ce4   [ "/l3vpn:vpn/l3vpn{volvo}" ]
ce5   [  ]
ce6   [  ]
ce7   [  ]
ce8   [ "/l3vpn:vpn/l3vpn{volvo}" ]
p0    [  ]
p1    [  ]
p2    [  ]
p3    [  ]
pe0   [ "/l3vpn:vpn/l3vpn{volvo}" ]
pe1   [  ]
pe2   [ "/l3vpn:vpn/l3vpn{volvo}" ]
pe3   [ "/l3vpn:vpn/l3vpn{volvo}" ]

Note that operational mode in the CLI was used above. Every service instance has an operational attribute that is maintained by the transaction manager and shows which device configuration it created. Furthermore every device configuration has backwards pointers to the corresponding service instances:

ncs(config)# show full-configuration devices device ce3 \
                    config | display service-meta-data
devices device ce3
 config
  ...
  /* Refcount: 1 */
  /* Backpointer: [ /l3vpn:vpn/l3vpn:l3vpn[l3vpn:name='ford'] ] */
  ios:interface GigabitEthernet0/2.100
   /* Refcount: 1 */
   description Link to PE / pe1 - GigabitEthernet0/0/0/5
   /* Refcount: 1 */
   encapsulation dot1Q 100
   /* Refcount: 1 */
   ip address 192.168.1.13 255.255.255.252
   /* Refcount: 1 */
   service-policy output ford
  exit

ncs(config)# show full-configuration devices device ce3 config \
                     | display curly-braces | display service-meta-data
...
ios:interface {
    GigabitEthernet 0/1;
    GigabitEthernet 0/10;
    GigabitEthernet 0/11;
    GigabitEthernet 0/12;
    GigabitEthernet 0/13;
    GigabitEthernet 0/14;
    GigabitEthernet 0/15;
    GigabitEthernet 0/16;
    GigabitEthernet 0/17;
    GigabitEthernet 0/18;
    GigabitEthernet 0/19;
    GigabitEthernet 0/2;
    /* Refcount: 1 */
    /* Backpointer: [ /l3vpn:vpn/l3vpn:l3vpn[l3vpn:name='ford'] ] */
    GigabitEthernet 0/2.100 {
        /* Refcount: 1 */
        description "Link to PE / pe1 - GigabitEthernet0/0/0/5";
        encapsulation {
            dot1Q {
                /* Refcount: 1 */
                vlan-id 100;
            }
        }
        ip {
            address {
                primary {
                    /* Refcount: 1 */
                    address 192.168.1.13;
                    /* Refcount: 1 */
                    mask    255.255.255.252;
                }
            }
        }
        service-policy {
            /* Refcount: 1 */
            output ford;
        }
    }

ncs(config)# show full-configuration devices device ce3 config \
                   | display service-meta-data | context-match Backpointer
devices device ce3
  /* Refcount: 1 */
  /* Backpointer: [ /l3vpn:vpn/l3vpn:l3vpn[l3vpn:name='ford'] ] */
  ios:interface GigabitEthernet0/2.100
devices device ce3
  /* Refcount: 2 */
  /* Backpointer: [ /l3vpn:vpn/l3vpn:l3vpn[l3vpn:name='ford'] ] */
  ios:interface GigabitEthernet0/5

The reference counter above makes sure that NSO will not delete shared resources until the last service instance is deleted. The context-match search is helpful, it displays the path to all matching configuration items.

As described in detail in the Device Manager the section called “Commit Queue” section the commit queue can be used to increase the transaction throughput. When the commit queue are for service activation the services will have states reflecting outstanding commit queue items.

We will now commit a vpn service using the commit queue and one device is down.

$ ncs-netsim stop ce0
DEVICE ce0 STOPPED

ncs(config)# show configuration
vpn l3vpn volvo
 as-number 65101
 endpoint branch-office1
  ce-device    ce1
  ce-interface GigabitEthernet0/11
  ip-network   10.7.7.0/24
  bandwidth    6000000
 !
 endpoint main-office
  ce-device    ce0
  ce-interface GigabitEthernet0/11
  ip-network   10.10.1.0/24
  bandwidth    12000000
 !
!

ncs# commit commit-queue async
commit-queue-id 10777927137
Commit complete.
ncs(config)# *** ALARM connection-failure: Failed to connect to device ce0: connection refused: Connection refused

This service is not provisioned fully in the network, since ce0 was down. It will stay in the queue either until the device starts responding or that an action is taken to remove the service or remove the item. The commit queue can be inspected. As shown below we see that we are waiting for ce0. Inspecting the queue item shows the outstanding configuration.

ncs# show devices commit-queue | notab
devices commit-queue queue-item 10777927137
 age              1934
 status           executing
 kilo-bytes-size  2
 devices          [ ce0 ce1 pe0 ]
 transient-errors [ ce0 ]
 is-atomic        true

ncs# show vpn l3vpn volvo commit-queue | notab
commit-queue queue-item 1498812003922
 status executing

The commit queue will constantly try to push the configuration towards the devices. The number of retry attempts and at what interval they occur can be configured.

ncs# show full-configuration devices global-settings commit-queue | details
devices global-settings commit-queue enabled-by-default false
devices global-settings commit-queue atomic true
devices global-settings commit-queue retry-timeout 30
devices global-settings commit-queue retry-attempts unlimited

If we start ce0 and inspect the queue we will see that the queue will finally be empty, and that the commit-queue status for the service is empty.

ncs# show devices commit-queue | notab
devices commit-queue queue-item 10777927137
 age              3357
 status           executing
 kilo-bytes-size  2
 devices          [ ce0 ce1 pe0 ]
 transient-errors [ ce0 ]
 is-atomic        true

ncs# show devices commit-queue | notab
devices commit-queue queue-item 10777927137
 age              3359
 status           executing
 kilo-bytes-size  2
 devices          [ ce0 ce1 pe0 ]
 is-atomic        true

ncs# show devices commit-queue
% No entries found.

ncs# show vpn l3vpn volvo commit-queue
% No entries found.

ncs# show devices commit-queue completed | notab
devices commit-queue completed queue-item 10777927137
 when               2015-02-09T16:48:17.915+00:00
 succeeded          true
 devices            [ ce0 ce1 pe0 ]
 completed          [ ce0 ce1 pe0 ]
 completed-services [ /l3vpn:vpn/l3vpn:l3vpn[l3vpn:name='volvo'] ]

In some scenarios it makes sense to remove the service configuration from the network but keep the representation of the service in NSO. This is called to un-deploy a service.

ncs# vpn l3vpn volvo check-sync
in-sync false
ncs# vpn l3vpn volvo re-deploy
ncs# vpn l3vpn volvo check-sync
in-sync true

Reactive Fastmap (RFM) services autonomously will do reactive-re-deploy until all "stages" of the service are completed. This implies that an RFM service normally is not completed when the initial create is committed. For the operator to understand that a RFM service has run to completion there must typically be some service specific operational data that can indicate this.

Plans are introduced to standardize the operational data that can show the progress of RFM service. This gives the user a standardized view of all RFM services and can directly answer the question weather a service instance has run to completion or not.

A plan consists of one or many component entries. Each component consists of two or many state entries where the state can be in status not-reached, reached or failed. A plan must have a component named "self" and can have other components with arbitrary names that have meaning for the implementing RFM service. A plan component must have a first state named "init" and a last state named "ready". In between "init" and "ready" a plan component can have additional state entries with arbitrary naming.

The purpose of the "self" component is to describe the main progress of the RFM service as a whole. Most importantly the "self" component last state named "ready" must have status "reached" if and only if the RFM service as a whole has completed. Other arbitrary components as well as states are added to the plan if they have meaning for the specific RFM service i.e more specific progress reporting.

A plan also defines an empty leaf failed which is set if and only if any state in any component has a status set to "failed". As such this is an aggregation to make it easy to verify if a RFM service is progressing without problems or not.

The following is an illustration of using the plan to report progress of an RFM service:

ncs# show vpn l3vpn volvo plan
NAME                    TYPE   STATE              STATUS       WHEN
------------------------------------------------------------------------------------
self                    self   init               reached      2016-04-08T09:22:40
                               ready              not-reached  -
endpoint-branch-office  l3vpn  init               reached      2016-04-08T09:22:40
                               qos-configured     reached      2016-04-08T09:22:40
                               ready              reached      2016-04-08T09:22:40
endpoint-head-office    l3vpn  init               reached      2016-04-08T09:22:40
                               pe-created         not-reached  -
                               ce-vpe-topo-added  not-reached  -
                               vpe-p0-topo-added  not-reached  -
                               qos-configured     not-reached  -
                               ready              not-reached  -

Plans were introduced to standardize the operational data that show progress of reactive fastmap (RFM) services. This gives the user a standardized view of all RFM services and can answer the question weather a service instance has run to completion or not. To keep track of the progress of plans service progress monitoring (SPM) is introduced. The idea with SPM is that time limits are put on the progress of plan states. To do so, a policy and a trigger is needed.

A policy defines what plan components and states needs to be in what status for the policy to be true. A policy also defines how long time it can be false without considered jeopardized and how long time it can be false without considered violated. Further it may define an action, that is called in case of a policy being jeopardized, violated or successful.

A trigger is used to associate a policy with a service and a component.

The following is an illustration of using a SPM to track progress of an RFM service, in this case the policy specifies that the self components ready state must be reached for the policy to be true:

ncs# show vpn l3vpn volvo service-progress-monitoring
                                                               JEOPARDY                       VIOLATION           SUCCESS
NAME  POLICY         START TIME           JEOPARDY TIME        RESULT    VIOLATION TIME       RESULT     STATUS   TIME
---------------------------------------------------------------------------------------------------------------------------
self  service-ready  2016-04-08T09:22:40  2016-04-08T09:22:40  -         2016-04-08T09:22:40  -          running  -