Containerized NSO

The Production Image is a production-ready NSO image for system-wide deployment and use. It is a pre-built Red Hat UBI-based NSO image created on System Install in Getting Started and available from the Cisco Software Download site.

Use the pre-built image as the base image in the container file (e.g., Dockerfile) and mount your own packages (such as NEDs and service packages) to run a final image for your production environment (see examples below).

The Development Image is a separate standalone NSO image with the necessary environment and software for building packages. It is also a pre-built Red Hat UBI-based image provided specifically to address the developer needs of building packages. The image is available as a signed package (e.g., nso-VERSION.container-image-dev.linux.ARCH.signed.bin) from the Cisco Software Download site. You can run the Development Image in different ways and a simple tool for defining and running multi-container Docker applications is Docker Compose (described below). The container provides the necessary environment to build custom packages.

The Development Image adds a few Linux packages that are useful for development, such as Ant, JDK, net-tools, pip, etc. Additional Linux packages can be added using, for example, the dnf command. The dnf list installed command lists all the installed packages.

To fetch and extract NSO images:

If you have NSO installed for production use using System Install, you can migrate to the Containerized NSO setup by following the instructions in this section. Migrating your Network Services Orchestrator (NSO) to a containerized setup can provide numerous benefits, including improved scalability, easier version management, and enhanced isolation of services.

The migration process is designed to ensure a smooth transition from a System-Installed NSO to a container-based deployment. Detailed steps guide you through preparing your existing environment, exporting the necessary configurations and state data, and importing them into your new containerized NSO instance. During the migration, consider the container runtime you plan to use, as this impacts the migration process.

The run-nso.sh script runs a check at startup to determine which ncs.conf file to use. The order of preference is as below:

If you need to perform operations before or after the ncs process is started in the Production container, you can use Python and/or Bash scripts to achieve this. Add the scripts to the $NCS_CONFIG_DIR/pre-ncs-start.d/ and $NCS_CONFIG_DIR/post-ncs-start.d/ directories to have the run-nso.sh script run them.

An admin user can be created on startup by the run script in the container. There are three environment variables that control the addition of an admin user:

As ADMIN_USERNAME already has a default value, only ADMIN_PASSWORD, or ADMIN_SSHKEY need to be set in order to create an admin user. For example:

docker run -itd --name cisco-nso -e ADMIN_PASSWORD=admin cisco-nso-prod:6.3

This can be useful when starting up a container in CI for testing or development purposes. It is typically not required in a production environment where there is a permanent CDB that already contains the required user accounts.

The default ncs.conf NSO configuration file does not enable any northbound interfaces, and no ports are exposed externally to the container. Ports can be exposed externally of the container when starting the container with the northbound interfaces and their ports enabled in ncs.conf.

The backup behavior of running NSO in vs. outside the container is largely the same, except that when running NSO in a container, the SSH and SSL certificates are not included in the backup produced by the ncs-backup script. This is different from running NSO outside a container where the default configuration path /etc/ncs is used to store the SSH and SSL certificates, i.e., /etc/ncs/ssh for SSH and /etc/ncs/ssl for SSL.

docker run -d --name cisco-nso -v NSO-vol:/nso -v NSO-log-vol:/log cisco-nso-prod:6.3

The NSO image /run-nso.sh script looks for an SSH host key named ssh_host_ed25519_key in the /nso/etc/ssh directory to be used by the NSO built-in SSH server for the CLI and NETCONF interfaces.

If an SSH host key exists, which is for a typical production setup stored in a persistent shared volume, it remains the same after restarts or upgrades of NSO. If no SSH host key exists, the script generates a private and public key.

In a high-availability (HA) setup, the host key is typically shared by all NSO nodes in the HA group and stored in a persistent shared volume. I.e., each NSO node does not generate its host key to avoid fetching the public host key after each failover from the new primary to access the primary's NSO CLI and NETCONF interfaces.

NSO expects to find a TLS certificate and key at /nso/ssl/cert/host.cert and /nso/ssl/cert/host.key respectively. Since the /nso path is usually on persistent shared volume for production setups, the certificate remains the same across restarts or upgrades.

If no certificate is present, one will be generated. It is a self-signed certificate valid for 30 days making it possible to use both in development and staging environments. It is not meant for the production environment. You should replace it with a properly signed certificate for production and it is encouraged to do so even for test and staging environments. Simply generate one and place it at the provided path, for example using the following, which is the command used to generate the temporary self-signed certificate:

openssl req -new -newkey rsa:4096 -x509 -sha256 -days 30 -nodes \
-out /nso/ssl/cert/host.cert -keyout /nso/ssl/cert/host.key \
-subj "/C=SE/ST=NA/L=/O=NSO/OU=WebUI/CN=Mr. Self-Signed"

The database in NSO, called CDB, uses YANG models as the schema for the database. It is only possible to store data in CDB according to the YANG models that define the schema.

If the YANG models are changed, particularly if the nodes are removed or renamed (rename is the removal of one leaf and an addition of another), any data in CDB for those leaves will also be removed. NSO normally warns about this when you attempt to load new packages, for example, request packages reload command refuses to reload the packages if the nodes in the YANG model have disappeared. You would then have to add the force argument, e.g., request packages reload force.

The base Production Image comes with a basic container health check. It uses ncs_cmd to get the state that NCS is currently in. Only the result status is observed to check if ncs_cmd was able to communicate with the ncs process. The result indicates if the ncs process is responding to IPC requests.

By default, the Linux kernel allows overcommit of memory. However, memory overcommit produces an unexpected and unreliable environment for NSO since the Linux Out Of Memory Killer, or OOM-killer, may terminate NSO without restarting it if the system is critically low on memory.

Also, when the OOM-killer terminates NSO, NSO will not produce a system dump file, and the debug information will be lost. Thus, it is strongly recommended that overcommit is disabled with Linux NSO production container hosts with an overcommit ratio of less than 100% (max).

See the section called “4. Run the Installer” in Getting Started for information on the memory overcommit recommendations for a Linux system hosting NSO production containers.

The /nso-run.sh script that starts NSO is executed as an ENTRYPOINT instruction and the CMD instruction can be used to provide arguments to the entrypoint-script. Another alternative is to use the EXTRA_ARGS variable to provide arguments. The /nso-run.sh script will first check the EXTRA_ARGS variable before the CMD instruction.

An example using docker run with the CMD instruction:

docker run --name nso -itd cisco-nso-prod:6.3 --with-package-reload \
--ignore-initial-validation

With the EXTRA_ARGS variable:

docker run --name nso \
-e EXTRA_ARGS='--with-package-reload --ignore-initial-validation' \
-itd cisco-nso-prod:6.3

An example using a Docker Compose file, compose.yaml, with the CMD instruction:

services:
    nso:
        image: cisco-nso-prod:6.3
        container_name: nso
        command:
            - --with-package-reload
            - --ignore-initial-validation

With the EXTRA_ARGS variable:

services:
    nso:
        image: cisco-nso-prod:6.3
        container_name: nso
        environment:
            - EXTRA_ARGS=--with-package-reload --ignore-initial-validation

This example shows how to run the standalone NSO Production Image using the Docker CLI.

The instructions and CLI examples used in this example are Docker-specific. If you are using a non-Docker container runtime, you will need to: fetch the NSO image from the Cisco software download site, then load and run the image with packages and networking, and finally log in to NSO CLI to run commands.

If you intend to run multiple images (i.e., both Production and Development), Docker Compose is a tool that simplifies defining and running multi-container Docker applications. See the example (the section called “Running the NSO Images using Docker Compose”) below for detailed instructions.

Steps

Follow the steps below to run the Production Image using Docker CLI:

This example describes how to upgrade your NSO to run a newer NSO version in container. The overall upgrade process is outlined in the steps below. In the example below, NSO is to be upgraded from version 6.2 to 6.3.

To upgrade your NSO version:

At this point, you only have one container that is running the desired version 6.3 and you do not need to uninstall the old NSO.

This example covers the necessary information to manifest the use of NSO images to compile packages and run NSO. Using Docker Compose is not a requirement, but a simple tool for defining and running a multi-container setup where you want to run both the Production and Development images in an efficient manner.

Packages

Packages used in this example are taken from the examples.ncs/development-guide/nano-services/netsim-sshkey example:

docker-compose.yaml - Docker Compose File Example

A basic Docker Compose file is shown in the example below. It describes the containers running on a machine:

Note that the packages use a shared volume in this simple example setup. In a more complex production environment, you may want to consider a dedicated redundant volume for your packages.

                version: '1.0'
                volumes:
                  NSO-1-rvol:

                networks:
                  NSO-1-net:

                services:
                  NSO-1:
                    image: cisco-nso-prod:6.3
                    container_name: nso1
                    profiles:
                      - prod
                    environment:
                      - EXTRA_ARGS=--with-package-reload
                      - ADMIN_USERNAME=admin
                      - ADMIN_PASSWORD=admin
                    networks:
                      - NSO-1-net
                    ports:
                      - "2024:2024"
                      - "8888:8888"
                    volumes:
                      - type: bind
                        source: /path/to/packages/NSO-1
                        target: /nso/run/packages
                      - type: bind
                        source: /path/to/log/NSO-1
                        target: /log
                      - type: volume
                        source: NSO-1-rvol
                        target: /nso
                    healthcheck:
                      test: ncs_cmd -c "wait-start 2"
                      interval: 5s
                      retries: 5
                      start_period: 10s
                      timeout: 10s

                  BUILD-NSO-PKGS:
                    image: cisco-nso-dev:6.3
                    container_name: build-nso-pkgs
                    network_mode: none
                    profiles:
                      - dev
                    volumes:
                      - type: bind
                        source: /path/to/packages/NSO-1
                        target: /nso/run/packages

                  EXAMPLE:
                    image: cisco-nso-prod:6.3
                    container_name: ex-netsim
                    profiles:
                      - example
                    networks:
                      - NSO-1-net
                    healthcheck:
                      test: test -f /nso-run-prod/etc/ncs.conf && ncs-netsim --dir /netsim is-alive ex0
                      interval: 5s
                      retries: 5
                      start_period: 10s
                      timeout: 10s
                    entrypoint: bash
                    command: -c 'rm -rf /netsim
                        && mkdir /netsim
                        && ncs-netsim --dir /netsim create-network /network-element 1 ex
                        && PYTHONPATH=/opt/ncs/current/src/ncs/pyapi ncs-netsim --dir
                            /netsim start
                        && mkdir -p /nso-run-prod/run/cdb
                        && echo "<devices xmlns=\"http://tail-f.com/ns/ncs\">
                            <authgroups><group><name>default</name>
                            <umap><local-user>admin</local-user>
                            <remote-name>admin</remote-name><remote-password>
                            admin</remote-password></umap></group>
                            </authgroups></devices>"
                            > /nso-run-prod/run/cdb/init1.xml
                        && ncs-netsim --dir /netsim ncs-xml-init >
                            /nso-run-prod/run/cdb/init2.xml
                        && sed -i.orig -e "s|127.0.0.1|ex-netsim|"
                            /nso-run-prod/run/cdb/init2.xml
                        && mkdir -p /nso-run-prod/etc
                        && sed -i.orig -e "s|</cli>|<style>c</style>
                            </cli>|" -e "/<ssh>/{n;s|<enabled>false
                                </enabled>|
                                <enabled>true</enabled>|}" defaults/ncs.conf
                        && sed -i.bak -e "/<local-authentication>/{n;s|
                            <enabled>false</enabled>|<enabled>true
                            </enabled>|}" defaults/ncs.conf
                        && sed "/<ssl>/{n;s|<enabled>false</enabled>|
                            <enabled>true</enabled>|}" defaults/ncs.conf
                            > /nso-run-prod/etc/ncs.conf
                        && mv defaults/ncs.conf.orig defaults/ncs.conf
                        && tail -f /dev/null'
                    volumes:
                      - type: bind
                        source: /path/to/packages/NSO-1/ne
                        target: /network-element
                      - type: volume
                        source: NSO-1-rvol
                        target: /nso-run-prod

Explanation of Docker Compose File

A description of noteworthy Compose file items is given below.

Steps

Follow the steps below to run the images using Docker Compose:

This example describes how to upgrade NSO when using Docker Compose.

Upgrade to a New Minor or Major Version

To upgrade to a new minor or major version, for example, from 6.2 to 6.3, follow the steps below:

Upgrade to a New Maintenance Release Version

To upgrade to a new maintenance release version, for example, to 6.3.1, follow the steps below: