NSO System Management

/ncs-config/webui/match-host-name to true.

/ncs-config/webui/server-name to the hostname of the server.

daemon-status You can see the NSO daemon mode, starting, phase0, phase1, started, stopping. The phase0 and phase1 modes are schema upgrade modes and will appear if you have upgraded any data-models.

version The NSO version.

smp Number of threads used by the daemon.

ha The High-Availability mode of the ncs daemon will show up here: secondary, primary, relay-secondary.

internal/callpoints Next section is call-points. Make sure that any validation points etc are registered. (The ncs-rfs-service-hook is an obsolete call-point, ignore this one).

Of special interest is of course the servicepoints. All your deployed service models should have a corresponding service-point. For example:

servicepoints:
  id=l3vpn-servicepoint daemonId=10 daemonName=ncs-dp-6-l3vpn:L3VPN
  id=nsr-servicepoint daemonId=11 daemonName=ncs-dp-7-nsd:NSRService
  id=vm-esc-servicepoint daemonId=12 daemonName=ncs-dp-8-vm-manager-
    esc:ServiceforVMstarting
  id=vnf-catalogue-esc daemonId=13 daemonName=ncs-dp-9-vnf-catalogue-
    esc:ESCVNFCatalogueService

internal/cdb The cdb section is important. Look for any locks. This might be a sign that a developer has taken a CDB lock without releasing it. The subscriber section is also important. A design pattern is to register subscribers to wait for something to change in NSO and then trigger an action. Reactive FASTMAP is designed around that. Validate that all expected subscribers are OK.

loaded-data-models The next section shows all namespaces and YANG modules that are loaded. If you for example are missing a service model, make sure it is really loaded.

cli, netconf, rest, snmp, webui All northbound agents like CLI, REST, NETCONF, SNMP etc are listed with their IP and port. So if you want to connect over REST for example, you can see the port number here.

patches Lists any installed patches.

upgrade-mode If the node is in upgrade mode, it is not possible to get any information from the system over NETCONF. Existing CLI sessions can get system information.

The daemon: ncs.smp: this is the ncs process running in the Erlang VM.

Java VM: com.tailf.ncs.NcsJVMLauncher: service applications implemented in Java runs in this VM. There are several options on how to start the Java VM, it can be monitored and started/restarted by NSO or by an external monitor. See ncs.conf(5) man page and the java-vm settings in the CLI.

Python VMs: NSO packages can be implemented in Python. The individual packages can be configured to run a VM each or share Python VM. Use the show python-vm status current to see current threads and show python-vm status start to see which threads where started at startup-time.

ncs.log : ncs daemon log. See the section called “Log messages and formats”. Can be configured to syslog.

ncserr.log.1, ncserr.log.idx, ncserr.log.siz: if the NSO daemon has a problem. this contains debug information relevant for support. The content can be displayed with "ncs --printlog ncserr.log".

audit.log: central audit log covering all northbound interfaces. See the section called “Log messages and formats” for formats. Can be configured to syslog.

localhost:8080.access: all HTTP requests to the daemon. This an access log for the embedded Web server. This file adheres to the Common Log Format, as defined by Apache and others. This log is not enabled by default and is not rotated, i.e., use logrotate(8). Can be configured to syslog.

devel.log: developer-log is a debug log for troubleshooting user-written code. This log is enabled by default and is not rotated, i.e., use logrotate(8). This log shall be used in combination with the java-vm or python-vm logs. The user code logs in the VM logs and corresponding library logs in devel.log. Disable this log in production systems. Can be configured to syslog. You can manage this log and set its logging level in ncs.conf.

    <developer-log>
      <enabled>true</enabled>
      <file>
        <name>${NCS_LOG_DIR}/devel.log</name>
        <enabled>false</enabled>
      </file>
      <syslog>
        <enabled>true</enabled>
      </syslog>
    </developer-log>
    <developer-log-level>trace</developer-log-level>

ncs-java-vm.log, ncs-python-vm.log: logger for code running in Java or Python VM, for example service applications. Developers writing Java and Python code use this log (in combination with devel.log) for debugging. Both Java and Python log levels can be set from their respective VM settings in, for example, the CLI.

admin@ncs(config)# python-vm logging level level-info
admin@ncs(config)# java-vm java-logging logger com.tailf.maapi level level-info

netconf.log, snmp.log: log for northbound agents. Can be configured to Syslog.

rollbackNNNNN: all NSO commits generates a corresponding rollback file. The maximum number of rollback files and file numbering can be configured in ncs.conf.

xpath.trace: XPATH is used in many places, for example XML templates. This log file shows the evaluation of all XPATH expressions and can be enabled in the ncs.conf.

    <xpathTraceLog>
      <enabled>true</enabled>
      <filename>${NCS_LOG_DIR}/xpath.trace</filename>
    </xpathTraceLog>

To debug XPATH for a template, use the pipe-target debug in the CLI instead.

admin@ncs(config)# commit | debug template

ned-cisco-ios-xr-pe1.trace (for example): if device trace is turned on a trace file will be created per device. The file location is not configured in ncs.conf but is configured when device trace is turned on, for example in the CLI.

admin@ncs(config)# devices device r0 trace pretty

Progress trace log: When a transaction or action is applied, NSO emits specific progress events. These events can be displayed and recorded in a number of different ways, either in CLI with the pipe-target details on a commit, or by writing it to a log file. You can read more about progress trace log in Progress Trace in Development Guide.

Transaction error log: log for collecting information on failed transactions that lead to either CDB boot error or a runtime transaction failure. The default is 'false' (disabled). More information about the log is available in the section called “CONFIGURATION PARAMETERS” in Manual Pages under logs/transaction-log.

Upgrade log: log containing information about CDB upgrade. The log is enabled by default and not rotated (i.e., use logrotate). With the NSO example set, the following examples populate the log in the logs/upgrade.log file: examples.ncs/development-guide/ned-upgrade/yang-revision, examples.ncs/development-guide/high-availability/upgrade-basic, examples.ncs/development-guide/high-availability/upgrade-cluster, and examples.ncs/getting-started/developing-with-ncs/14-upgrade-service. More information about the log is available in the section called “CONFIGURATION PARAMETERS” in Manual Pages under logs/upgrade-log.

Exit codes 1 and 19 mean that an internal error has occurred. A text message should be in the logs, or if the error occurred at startup before logging had been activated, on standard error (standard output if NSO was started with --foreground --verbose). Generally the message will only be meaningful to the NSO developers, and an internal error should always be reported to support.

Exit codes 2 and 3 are only used for the ncs "control commands" (see the section COMMUNICATING WITH NCS in the ncs(1) in Manual Pages manual page), and mean that the command failed due to timeout. Code 2 is used when the initial connect to NSO didn't succeed within 5 seconds (or the TryTime if given), while code 3 means that the NSO daemon did not complete the command within the time given by the --timeout option.

Exit code 10 means that one of the init files in the CDB directory was faulty in some way. Further information in the log.

Exit code 11 means that the CDB configuration was changed in an unsupported way. This will only happen when an existing database is detected, which was created with another configuration than the current in ncs.conf.

Exit code 13 means that the schema change caused an upgrade, but for some reason the upgrade failed. Details are in the log. The way to recover from this situation is either to correct the problem or to re-install the old schema (fxs) files.

Exit code 14 means that the schema change caused an upgrade, but for some reason the upgrade failed, corrupting the database in the process. This is rare and usually caused by a bug. To recover, either start from an empty database with the new schema, or re-install the old schema files and apply a backup.

Exit code 15 means that A.cdb or C.cdb is corrupt in a non-recoverable way. Remove the files and re-start using a backup or init files.

Exit code 16 means that CDB ran into an unrecoverable file-error (such as running out of space on the device while performing journal compaction).

Exit code 20 means that NSO failed to bind a socket.

Exit code 21 means that some NSO configuration file is faulty. More information in the logs.

Exit code 22 indicates a NSO installation related problem, e.g. that the user does not have read access to some library files, or that some file is missing.

Out of memory: If NSO is unable to allocate memory, it will exit by calling abort(3). This will generate an exit code as for reception of the SIGABRT signal - e.g. if NSO is started from a shell script, it will see 134 as exit code (128 + the signal number).

Out of file descriptors for accept(2): If NSO fails to accept a TCP connection due to lack of file descriptors, it will log this and then exit with code 25. To avoid this problem, make sure that the process and system-wide file descriptor limits are set high enough, and if needed configure session limits in ncs.conf.

In many Linux systems the default limit is 1024, but if we, for example, assume that there are 4 northbound interface ports, CLI, RESTCONF, SNMP, WebUI/JSON-RPC, or similar, plus a few hundreds of IPC ports, x 1024 == 5120. But one might as well use the next power of two, 8192, to be on the safe side.

Several application issues can contribute to consuming extra ports. In the scope of a NSO application that could, for example, be a script application that invokes CLI command or a callback daemon application that does not close the connection socket as they should.

A commonly used command for changing the maximum number of open file descriptors is ulimit -n [limit]. Commands such as netstat and lsof can be useful to debug file descriptor related issues.