NSO Deployment

If the cfs-s node reboots, NSO will start from the /etc boot scripts. The HA component cannot automatically decide what to do though. It will await an explicit operator command. After reboot, we will see:

  klacke@cfs-s> show status ncs-state ha
  mode none;
  [ok][2016-01-07 18:36:41]
  klacke@cfs-s> show status ha
  member cfs-m {
      current-ha-role unknown;
  }
  member cfs-s {
      current-ha-role unknown;
  }

On the designated secondary node, and the preferred primary node cfs-m will show:

  klacke@cfs-m> show status ha
  member cfs-m {
      current-ha-role master;
  }
  member cfs-s {
      current-ha-role unknown;
  }

To remedy this, the operator must once again activate HA. In suffices to to it on cfs-s, but we can in this case safely do it on both nodes, even doing it using the nct ha command. Re-activating HA on cfs-s will ensure that

This is the interesting fail over scenario. Powering off the cfs-m primary node, we see the following on cfs-s:

This is a critical moment, HA has failed over. When the original primary cfs-m restarts, the operator MUST manually decide what to do. Restarting cfs-m we get:

  klacke@cfs-m> show status ha
  member cfs-m {
      current-ha-role unknown;
  }
  member cfs-s {
      current-ha-role unknown;
  }

If we now activate the original primary, it will resume it's former primary role. Since cfs-s already is primary, this will be a mistake. Instead we must

  klacke@cfs-m> request ha commands role-override role slave
  status override
  [ok][2016-01-11 14:28:27]
  klacke@cfs-m> request ha commands activate
  status activated
  [ok][2016-01-11 14:28:42]
  klacke@cfs-m> show status ha
      member cfs-m {
  current-ha-role slave;
  }
  member cfs-s {
      current-ha-role master;
  }

This means that all config from cfs-s will be copied back to cfs-m. Once HA is once again established, we can easily go back to original situation by executing:

  klacke@cfs-m> request ha commands role-revert

on both nodes. This is recommended in order to have the running situation as normal as possible.

Note: This is indeed a critical operation. It's actually possible to loose all or some data here. For example, assume that the original primary cfs-m was down for a period of time, the following sequence of events/commands will loose data.

The above sequence of events/commands loose all provisioning requests between t0 and t1

The final part of configuring HA is enabling either IP layer 2 VIP (Virtual IP) support or IP layer 3 BGP anycast fail over. Here we will describe layer 2 VIP configuration, details about anycast setup can be found in the tailf-hcc documentation.

We modify the HA configuration so that it looks as:

  klacke@cfs-m% show ha
  token         xyz;
  interval      4;
  failure-limit 10;
  vip {
      address 10.147.41.253;
  }
  member cfs-m {
      address         10.147.40.190;
      default-ha-role master;
      vip-interface   eth0;
  }
  member cfs-s {
      address         10.147.40.77;
      default-ha-role slave;
      failover-master true;
      vip-interface   eth0;
  }

Whenever a node is primary, it will also bring up a VIP.

  $ ifconfig eth0:ncsvip
  eth0:ncsvip Link encap:Ethernet  HWaddr 08:00:27:0c:c3:48
  inet addr:10.147.41.253  Bcast:10.147.41.255  Mask:255.255.254.0
  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

The purpose of the VIP is to have northbound systems, northbound provisioning systems, activate NSO services through the VIP which will always be reachable as long as one NSO system is still up.

Referring to previous discussion above on manual activation, if the operator reactivates HA on the designated primary after a primary reboot, we will end up with two primaries, both activating the VIP using gratuitous ARP. This must be avoided at all costs - thus HA activation must be done with care. It cannot be automated.

There are multiple aspects that you should consider before starting with the actual upgrade procedure. While the development team tries to provide as much compatibility between software releases as possible, all incompatible changes cannot always be avoided. For example, when a deviation from an RFC standard is found and resolved, it may break clients that depend on the non-standard behavior. For this reason, a distinction is made between a maintenance and a major NSO upgrade.

A maintenance NSO upgrade is within the same branch, that is, when the first two version numbers stay the same (x.y in the x.y.z NSO version). An example is upgrading from version 5.6.1 to 5.6.2. In the case of a maintenance upgrade, the NSO release contains only corrections and minor enhancements, minimizing the changes. It includes binary compatibility for packages, which means there's no need to recompile the .fxs files for a maintenance upgrade.

Correspondingly, when the first or second number in the version changes, that is called a full or major upgrade. For example, upgrading version 5.6.1 to 5.7 is a major, non-maintenance upgrade. Due to new features, packages must be recompiled and some incompatibilities could manifest.

In addition to the above, a package upgrade is when you replace a package, such as a NED or a service package, with a newer version. Sometimes, when package changes aren't too big, it's possible to supply the new packages as part of the NSO upgrade, but this approach brings additional complexity. Instead, package upgrade and NSO upgrade should in general be performed as separate actions and are covered as such.

To avoid surprises during any upgrade, first ensure the following:

In case it turns out any of the packages are incompatible or cannot be simply compiled again, you will need to contact the package developers for an updated or recompiled version. For an official Cisco-supplied package, it's recommended that you always obtain a pre-compiled version if it is available for the target NSO release, instead of compiling the package yourself.

Additional preparation steps may be required based on the upgrade and the actual setup, such as when using the Layered Service Architecture (LSA) feature. In particular, for a major NSO upgrade in a multi-version LSA cluster, ensure that the new version supports the other members of the cluster and follow the additional steps outlined in Setting up LSA deployments in NSO Layered Service Architecture.

If you use the High Availability (HA) feature, the upgrade consists of multiple steps on different nodes. To avoid mistakes, you are encouraged to script the process, for which you will need to set up and verify access to all NSO instances with either ssh, nct, or some other remote management command.

Please be aware that NSO 5 introduced major changes in device model handling. See the NSO CDM Migration Guide if upgrading from a previous release.

Likewise, NSO 5.3 added support for 256 bit AES encrypted strings, requiring the AES256CFB128 key in the ncs.conf configuration. You can generate one with the openssl rand -hex 32 or similar command. Alternatively, if you use an external command for providing keys, make sure that it includes a value for an AES256CFB128_KEY in the output.

Finally, regardless of the upgrade type, make sure that you have a working backup and can easily restore the previous configuration if needed, as described in the section called “Backup and restore”.

The upgrade of a single NSO instance requires the following steps:

The following steps suppose that you are upgrading to the 5.7 release. They pertain to a system install of NSO and you must perform them with Super User privileges. As a best practice, always create a backup before trying to upgrade.

# ncs-backup

For the upgrade itself, you must first download to the host and install the new NSO release.

# sh nso-5.7.linux.x86_64.installer.bin --system-install

Then, you stop the currently running server with the help of the init.d script or an equivalent command, relevant to your system.

# /etc/init.d/ncs stop
Stopping ncs: .

Next, you update the symbolic link for the currently selected version to point to the newly installed one, 5.7 in this case.

# cd /opt/ncs
# rm -f current
# ln -s ncs-5.7 current

While seldom necessary, at this point you would also update the /etc/ncs/ncs.conf file.

Now, make sure that the /var/opt/ncs/packages/ directory has packages that are appropriate for the new version. For a maintenance upgrade, it should be possible to continue using the same packages. But for a major upgrade, you must normally rebuild the packages or use packages pre-built for the new version. It is very important that you ensure this directory contains the exact same version of each existing package, compiled for the new release, and nothing else.

As a best practice, the available packages are kept in /opt/ncs/packages/ and /var/opt/ncs/packages/ only contains symbolic links. In this case, to identify the release for which they were compiled for, the package file names all start with the corresponding NSO version. Then, you only need to rearrange the symbolic links in the /var/opt/ncs/packages/ directory.

# cd /var/opt/ncs/packages/
# rm -f *
# for pkg in /opt/ncs/packages/ncs-5.7-*; do ln -s $pkg; done

Please note that the above package naming scheme is neither required nor enforced. If your package filesystem names differ from it, you will need to adjust the preceding command accordingly.

Finally, you start the new version of the NSO server with the package reload flag set.

# /etc/init.d/ncs start-with-package-reload
Starting ncs: .

NSO will perform the necessary data upgrade automatically. If you have changed or removed any packages, this process may fail. In that case, ensure that the correct versions of all packages are present in /var/opt/ncs/packages/ and retry the preceding command.

Also note that with many packages or data entries in the CDB this process could take more than 90 seconds and result in the following error message being reported:

Starting ncs (via systemctl): Job for ncs.service failed
because a timeout was exceeded. See "systemctl status
ncs.service" and "journalctl -xe" for details. [FAILED]

That does not imply that NSO failed to start, just that it took longer than 90 seconds. It is recommended you wait some additional time before verifying.

It is imperative you have a working copy of data available from which you can restore. That is why you must always create a backup before starting an upgrade. Only a backup guarantees that you can rerun the upgrade or back out of it, should it be necessary.

The same steps can also be used to restore data on a new, similar host if OS of the initial host becomes corrupted beyond repair.

First, stop the NSO process if it is running.

# /etc/init.d/ncs stop
Stopping ncs: .

Verify and, if necessary, revert the symbolic link in /opt/ncs/ to point to the initial NSO release.

# cd /opt/ncs
# ls -l current
# ln -s ncs-VERSION current

In the exceptional case where the initial version installation was removed or damaged, you will need to re-install it first and redo the step above.

Verify if the correct (initial) version of NSO is being used.

# ncs --version

Next, restore the backup.

# ncs-backup --restore

Finally, start the NSO server and verify the restore was successful.

# /etc/init.d/ncs start
Starting ncs: .

Upgrading NSO in a highly available (HA) setup is a staged process. It entails running various commands across multiple NSO instances at different times.

The procedure is almost the same for a maintenance and major NSO upgrade. The difference is that a major upgrade requires the replacement of packages with recompiled ones. Still, a maintenance upgrade is often perceived as easier because there are fewer changes in the product.

In addition, this same process can also be used for only upgrading the packages.

The stages of the upgrade are:

The main thing to note is that all packages must match the NSO release. If they do not, the upgrade will fail.

In the case of a major upgrade, you must recompile the packages for the new version. It is highly recommended that you use pre-compiled packages and do not compile them during this upgrade procedure since the compilation can prove nontrivial, and the production hosts may lack all the required (development) tooling. You should use a naming scheme to distinguish between packages compiled for different NSO versions. A good option is for package file names to start with the ncs-MAJORVERSION- prefix for a given major NSO version. This ensures multiple packages can co-exist in the /opt/ncs/packages folder, and the NSO version they can be used with becomes obvious.

The following is a transcript of a sample upgrade procedure, showing the commands for each step described above, in a 2-node HA setup, with nodes in their initial designated state.

<switch to designated primary CLI>
admin@ncs# show high-availability status mode
high-availability status mode primary
admin@ncs# high-availability read-only mode true

<switch to designated secondary CLI>
admin@ncs# show high-availability status mode
high-availability status mode secondary
admin@ncs# high-availability read-only mode true

<switch to designated primary shell>
# ncs-backup

<switch to designated secondary shell>
# ncs-backup

<switch to designated primary CLI>
admin@ncs# high-availability disable

<switch to designated secondary CLI>
admin@ncs# high-availability be-master

<switch to designated primary shell>
# <upgrade node>
# /etc/init.d/ncs restart-with-package-reload

<switch to designated secondary CLI>
admin@ncs# high-availability disable

<switch to designated primary CLI>
admin@ncs# high-availability enable

<switch to designated secondary shell>
# <upgrade node>
# /etc/init.d/ncs restart-with-package-reload

<switch to designated secondary CLI>
admin@ncs# high-availability enable

Scripting is a recommended way to upgrade the NSO version of an HA cluster. The following example script shows the required commands and can serve as a basis for your own customized upgrade script. In particular, the script requires a specific package naming convention above, and you may need to tailor it to your environment. In addition, it expects the new release version and the designated primary and secondary node addresses as the arguments. The recompiled packages are read from the packages-MAJORVERSION/ directory.

For the below example script we configured our primary and secondary nodes with their nominal roles that they assume at startup and when HA is enabled. Automatic failover is also enabled so that the secondary will assume the primary role if the primary node goes down.

<config xmlns="http://tail-f.com/ns/config/1.0">
  <high-availability xmlns="http://tail-f.com/ns/ncs">
    <ha-node>
      <id>n1</id>
      <nominal-role>master</nominal-role>
    </ha-node>
    <ha-node>
      <id>n2</id>
      <nominal-role>slave</nominal-role>
      <failover-master>true</failover-master>
    </ha-node>
    <settings>
      <enable-failover>true</enable-failover>
      <start-up>
        <assume-nominal-role>true</assume-nominal-role>
        <join-ha>true</join-ha>
      </start-up>
    </settings>
  </high-availability>
</config>

#!/bin/bash
set -ex

vsn=$1
primary=$2
secondary=$3
installer_file=nso-${vsn}.linux.x86_64.installer.bin
pkg_vsn=$(echo $vsn | sed -e 's/^\([0-9]\+\.[0-9]\+\).*/\1/')
pkg_dir="packages-${pkg_vsn}"

function on_primary() { ssh $primary "$@" ; }
function on_secondary() { ssh $secondary "$@" ; }
function on_primary_cli() { ssh -p 2024 $primary "$@" ; }
function on_secondary_cli() { ssh -p 2024 $secondary "$@" ; }

function upgrade_nso() {
    target=$1
    scp $installer_file $target:
    ssh $target "sh $installer_file --system-install --non-interactive"
    ssh $target "rm -f /opt/ncs/current && \
                 ln -s /opt/ncs/ncs-${vsn} /opt/ncs/current"
}
function upgrade_packages() {
    target=$1
    do_pkgs=$(ls "${pkg_dir}/" || echo "")
    if [ -n "${do_pkgs}" ] ; then
        cd ${pkg_dir}
        ssh $target 'rm -rf /var/opt/ncs/packages/*'
        for p in ncs-${pkg_vsn}-*.gz; do
            scp $p $target:/opt/ncs/packages/
            ssh $target "ln -s /opt/ncs/packages/$p /var/opt/ncs/packages/"
        done
        cd -
    fi
}

# Perform the actual procedure

on_primary_cli 'request high-availability read-only mode true'
on_secondary_cli 'request high-availability read-only mode true'

on_primary 'ncs-backup'
on_secondary 'ncs-backup'

on_primary_cli 'request high-availability disable'
on_secondary_cli 'request high-availability be-master'
upgrade_nso $primary
upgrade_packages $primary
on_primary '/etc/init.d/ncs restart-with-package-reload'

on_secondary_cli 'request high-availability disable'
on_primary_cli 'request high-availability enable'
upgrade_nso $secondary
upgrade_packages $secondary
on_secondary '/etc/init.d/ncs restart-with-package-reload'

on_secondary_cli 'request high-availability enable'

Once the script completes, it is paramount that you manually verify the outcome. First, check that the HA is enabled by using the show high-availability command on the CLI of each node. Then connect to the designated secondaries and ensure they have the complete latest copy of the data, synchronized from the primaries.

The described upgrade procedure is for an HA pair. The nodes are expected to have initially assigned (nominal) roles, and the procedure ensures that is the case at the end. For a 3-node consensus setup, first disable the HA on the third (non-fail-over) node, perform the described procedure, and finally upgrade the 3rd node as well.

After the primary node is upgraded and restarted, the read-only mode is automatically disabled. This allows the primary node to start processing writes, minimizing downtime. However, there is no HA. Should the primary fail at this point or you need to revert to a pre-upgrade backup, the new writes would be lost. To avoid this scenario, again enable read-only mode on the primary after re-enabling HA. Then disable read-only mode only after successfully upgrading and reconnecting the secondary.

To further reduce time spent upgrading, you can customize the script to install the new NSO release and copy packages beforehand. Then, you only need to switch the symbolic links and restart the NSO process to use the new version.

You can use the same script for a maintenance upgrade as-is, with an empty packages-MAJORVERSION directory, or remove the upgrade_packages calls from the script.

Example implementations that use scripts to upgrade a 2- and 3-node setup using CLI/MAAPI or RESTCONF are available in the NSO example set under examples.ncs/development-guide/high-availability.

If you do not wish to automate the upgrade process, you will need to follow the instructions from the section called “Single Instance Upgrade” and transfer the required files to each host manually. Additional information on HA is available in High Availability. However, you can run the high-availability actions from the preceding script on the NSO CLI as-is. In this case, please take special care on which host you perform each command, as it can be easy to mix them up.

Package upgrades are frequent and routine in development but require the same care as NSO upgrades in the production environment. The reason is that the new packages may contain an updated YANG model, resulting in a data upgrade process similar to version upgrade. So, if a package is removed or uninstalled and a replacement is not provided, package-specific data, such as service instance data, will be removed as well.

In a single node environment, the procedure is straightforward. Create a backup with the ncs-backup command and ensure the new package is compiled for the current NSO version and available under the /opt/ncs/packages directory. Then either manually rearrange the symbolic links in the /var/opt/ncs/packages directory or use the software packages install command in the NSO CLI. Finally, invoke the packages reload command. For example:

# ncs-backup
INFO  Backup /var/opt/ncs/backups/ncs-5.7@2022-01-21T10:34:42.backup.gz created successfully
# ls /opt/ncs/packages
ncs-5.7-router-nc-1.0 ncs-5.7-router-nc-1.0.2
# ncs_cli -C
admin@ncs# software packages install package router-nc-1.0.2 replace-existing
installed ncs-5.7-router-nc-1.0.2
admin@ncs# packages reload

>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.
reload-result {
    package router-nc-1.0.2
    result true
}

On the other hand, upgrading packages in an HA setup is a staged process. Broadly, it follows the same sequence of steps as upgrading the NSO and should be scripted for the same reasons. The difference is that you must explicitly uninstall the old packages and install the new ones.

Next you will find a description of an upgrade procedure for an HA pair. It is expected that all nodes are in their assigned (nominal) roles initially and the procedure ensures that's the case at the end. For a 3-node consensus setup, you must first disconnect the third (non-fail-over) node, perform the described procedure, and finally upgrade the 3rd node as well.

After backing up all the nodes and disabling writes, switch over the HA to the secondary node, allowing you to perform the necessary work on the primary. Having disabled the HA on the primary node, execute the following instructions on the primary.

Transfer the new packages into /opt/ncs/packages with the help of the scp command or some other way. Select the correct packages by manually rearranging the symlinks in the /var/opt/ncs/packages folder or using the software packages install/deinstall commands in the CLI. Lastly, execute the packages reload command.

After verifying the node was successfully upgraded, switch the HA back to the upgraded primary, by disabling the HA on the designated secondary and enabling it on the designated primary. Then, repeat the transfer and upgrade of the packages on the designated secondary from the previous paragraph. Finally, reactivate the HA on the designated secondary and disable read-only mode.

The following example script codifies this procedure for an upgrade of a single package. Please customize it to your specific needs and environment.

#!/bin/bash
set -ex

primary=$1
secondary=$2
oldpkg=$3
newpkg=$4

function on_primary() { ssh $primary "$@" ; }
function on_secondary() { ssh $secondary "$@" ; }
function on_primary_cli() { ssh -p 2024 $primary "$@" ; }
function on_secondary_cli() { ssh -p 2024 $secondary "$@" ; }


on_primary_cli 'request high-availability read-only mode true'
on_secondary_cli 'request high-availability read-only mode true'

on_primary 'ncs-backup'
on_secondary 'ncs-backup'

on_primary_cli 'request high-availability disable'
on_secondary_cli 'request high-availability be-master'
scp ${newpkg}.tar.gz $primary:/opt/ncs/packages/
on_primary_cli "request software packages deinstall package ${oldpkg}"
on_primary_cli "request software packages install package ${newpkg}"
on_primary_cli 'request packages reload'

on_secondary_cli 'request high-availability disable'
on_primary_cli 'request high-availability enable'
scp ${newpkg}.tar.gz $secondary:/opt/ncs/packages/
on_secondary_cli "request software packages deinstall package ${oldpkg}"
on_secondary_cli "request software packages install package ${newpkg}"
on_secondary_cli 'request packages reload'

on_secondary_cli 'request high-availability enable'

You can extend the script to handle multiple packages in one go, making it more efficient. In that case, you should also consider using the request packages ha sync CLI command to further optimize the process. This command distributes all available packages from the current primary node to secondary nodes but does not install them. The command does not perform the sync on the node with none role.

The script uses the packages reload command to load new data models into NSO instead of restarting the server process. It is considerably more efficient and the time difference to upgrade can be considerable if the amount of data in CDB is huge.

In some cases, NSO may give warnings when the upgrade looks "suspicious." For more information on this please see the section called “Loading Packages”. If you understand the implications and are willing to risk losing data, use the force option with packages reload or set the NCS_RELOAD_PACKAGES environment variable to force when restarting NSO. It will force NSO to ignore warnings and proceed with the upgrade. In general, this is not recommended.

In addition, you must take special care for NED upgrades because services depend on them. Since NSO 5 introduced the CDM feature, which allows loading multiple versions of a NED, a major NED upgrade requires a procedure involving the migrate action.

When a NED contains nontrivial YANG model changes, that is called a major NED upgrade. The ned-id changes and also the first or the second number in the NED version changes, since NEDs follow the same versioning scheme as NSO. In this case, you cannot simply replace the package, as you would for a maintenance or patch NED release. Instead, you must load (add) the new NED package alongside the old one and perform the migration.

Migration uses the /ncs:devices/device/migrate action to change the ned-id of a single device or a group of devices. It does not affect the actual network device, except possibly reading from it. So, the migration does not have to be performed as part of the package upgrade procedure described above but can be done later, during normal operations. The details are described in the section called “NED Migration”. Once the migration is complete, you can remove the old NED by performing another package upgrade, where you “deinstall” the old NED package. It can be done straight after the migration or as part of the next upgrade cycle.

NSO has the ability to install emergency patches during runtime. These are delivered in the form of .beam files. You must copy the files into the /opt/ncs/current/lib/ncs/patches/ folder and load them with the ncs-state patches load-modules command.

The NSO system install that you have performed on your 4 hosts also installs good defaults for logrotate. Inspect /etc/logrotate.d/ncs and ensure that the settings are what you want. Note: The NSO error logs, i.e the files /var/log/ncs/ncserr.log* are internally rotated by NSO and MUST not be rotated by logrotate.

A crucial tool for debugging NSO installations are NED logs. These logs are very verbose and are for debugging only. Do not have these logs enabled in production. Note that everything, including potentially sensitive data, is logged. No filtering is done. The NED trace logs are controlled through in CLI under: /device/global-settings/trace. It's also possible to control the NED trace on a per device basis under /devices/device[name='x']/trace.

There are 3 different levels of trace, and for various historic reasons you usually want different settings depending on the device type.

Thus, it's usually not good enough to just control the NED trace from /devices/global-settings/trace.

User application Java logs are written to /var/log/ncs/ncs-java-vm.log. The level of logging from Java code is controlled on a per Java package basis. For example, if you want to increase the level of logging on e.g the tailf-hcc code, you need to look into the code and find out the name of the corresponding Java package. Unpacking the tailf-hcc tar.gz package, you see in the tailf-hcc/src/java/src/com/tailf/ns/tailfHcc/TcmApp.java file that the package is called com.tailf.ns.tailfHcc. You can then do:

  klacke@cfs-s% show java-vm java-logging
  logger com.tailf.ns.tailfHcc {
      level level-all;
  }

The internal NSO log resides at /var/log/ncs/ncserr.*. The log is written in a binary format, to view the internal error log run the following command:

  $ ncs --printlog /var/log/ncs/ncserr.log

The nct get-logs command grabs all logs from all hosts. This is good when collecting data from the system.

The REST api can be used to view the NSO alarm table. NSO alarms are not events, whenever an NSO alarm is created - an SNMP trap is also sent (assuming that you have configured a proper SNMP target) All alarms require operator invention. Thus, a monitoring tool should also GET the NSO alarm table.

curl -k -u klacke:PASSW https://cfs-m:8888/api/operational/alarms/alarm-list -X GET

Whenever there are new alarms, an operator MUST take a look.