The NSO Java VM

The NSO Java VM is the execution container for all java classes supplied by deployed NSO packages. The classes, and other resources, are structured in jar files and the specific use of these classes are described in the component tag in respective package-meta-data.xml file. Also as a framework, it starts and control other utilities for the use of these components. To accomplish this a main class com.tailf.ncs.NcsMain, implementing the Runnable interface, is started as a thread. This thread can be the main thread (running in a java main()) or be embedded into another java program.

When the NcsMain thread starts it establishes a socket connection towards NSO. This is called the NSO Java VM control socket. It is the responsibility of NcsMain to respond to command requests from NSO and pass these commands as events to the underlying finite state machine (FSM). The NcsMain FSM will execute all actions as requested by NSO. This include class loading and instantiation as well as registration and start of services, NEDs etc.

Figure 21. NSO Service Manager

When NSO detects the control socket connect from NSO Java VM, it starts an initialization process.

See the section called “Debugging Startup” for tips on customizing startup behavior and debugging problems when the Java VM fails to start

The file tailf-ncs-java-vm.yang defines the java-vm container which, along with ncs.conf, is the entry point for controlling the NSO Java VM functionality. Study the content of the YANG model in Example 18, “The Java VM YANG model”. For a full explanation of all the configuration data, look at the YANG file and man ncs.conf.

Many of the nodes beneath java-vm are by default invisible due to a hidden attribute. In order to make everything beneath java-vm visible in the CLI, two steps are required. First the following XML snippet must be added to ncs.conf:

<hide-group>
  <name>debug</name>
</hide-group>

Now the unhide command may be used in the CLI session:

admin@ncs(config)# unhide debug
admin@ncs(config)#

        > yanger -f tree tailf-ncs-java-vm.yang
          submodule: tailf-ncs-java-vm (belongs-to tailf-ncs)
  +--rw java-vm
     +--rw stdout-capture
     |  +--rw enabled?   boolean
     |  +--rw file?      string
     |  +--rw stdout?    empty
     +--rw connect-time?                     uint32
     +--rw initialization-time?              uint32
     +--rw synchronization-timeout-action?   enumeration
     +--rw exception-error-message
     |  +--rw verbosity?   error-verbosity-type
     +--rw java-logging
     |  +--rw logger* [logger-name]
     |     +--rw logger-name    string
     |     +--rw level          log-level-type
     +--rw jmx!
     |  +--rw jndi-address?   inet:ip-address
     |  +--rw jndi-port?      inet:port-number
     |  +--rw jmx-address?    inet:ip-address
     |  +--rw jmx-port?       inet:port-number
     +--ro start-status?                     enumeration
     +--ro status?                           enumeration
     +---x stop
     |  +--ro output
     |     +--ro result?   string
     +---x start
     |  +--ro output
     |     +--ro result?   string
     +---x restart
        +--ro output
           +--ro result?   string

Each NSO package will have a specific java classloader instance that loads its private jar classes. These package classloaders will refer to a single shared classloader instance as its parent. The shared classloader will load all shared jar classes for all deployed NSO packages.

The purpose of this is first to keep integrity between packages which should not have access to each others classes, other than the ones that are contained in the shared jars. Secondly, this way it is possible to hot redeploy the private jars and classes of a specific package while keeping other packages in a run state.

Should this class loading scheme not be desired, it is possible to suppress it by starting the NSO Java VM with the system property TAILF_CLASSLOADER set to false.

        java -DTAILF_CLASSLOADER=false ...

This will force NSO Java VM to use the standard java system classloader. For this to work, all jars from all deployed NSO packages needs to be part of the classpath. The drawback of this is that all classes will be globally accessible and hot redeploy will have no effect.

There are 4 types of components that the NSO Java VM can handle:

In some situations several NSO packages are expected to use the same code base, e.g. when third party libraries are used or the code is structured with some common parts. Instead of duplicate jars in several NSO packages it is possible to create a new NSO package, add these jars to the shared-jar directory and let the package-meta-data.xml file contain no component definitions at all. The NSO Java VM will load these shared jars and these will be accessible from all other NSO packages.

Inside the NSO Java VM each component type have a specific Component Manager. The responsibility of these Managers are to manage a set of component classes for each NSO package. The Component Manager act as a FSM that controls when a component should be registered, started, stopped etc.

Figure 22. Component Managers

For instance the DpMuxManager controls all callback implementations (services, actions, data providers etc). It can load, register, start and stop such callback implementations.

NEDs can be of type netconf, snmp, cli or generic. Only the cli and generic types are relevant for the NSO Java VM because these are the ones that have a java implementation. Normally these NED components comes in self contained and prefabricated NSO packages for some equipment or class of equipment. It is however possible to tailor make NEDs for any protocol. For more information on this see NED Development and NED Development

Callbacks are the collective name for a number of different functions that can be implemented in java. One of the most important is the service callbacks, but also actions, transaction control and data provision callbacks are in common use in an NSO implementation. For more on how to program callback using the Dp API. See the section called “DP API”

For programs that are none of the above types but still need to access NSO as a daemon process it is possible to use the ApplicationComponent java interface. The ApplicationComponent interface expects the implementing classes to implement a init(), finish() and a run() method.

The NSO Java VM will start each class in a separate thread. The init() is called prior to the thread is started. The run() runs in a thread similar to the run() method in the standard java Runnable interface. The finish() method is called when the NSO Java VM wants the application thread to stop. It is the responsibility of the programmer to stop the application thread i.e stop the execution in the run() method when finish() is called. Note, that making the thread stop when finish() is called is important so that the NSO Java VM will not be hanging at a STOP_VM request.

package com.tailf.ncs;

/**
 * User defined Applications should implement this interface that
 * extends Runnable, hence also the run() method has to be implemented.
 * These applications are registered as components of type
 * "application" in a Ncs packages.
 *
 * Ncs Java VM will start this application in a separate thread.
 * The init() method is called before the thread is started.
 * The finish() method is expected to stop the thread. Hence stopping
 * the thread is user responsibility
 *
 */
public interface ApplicationComponent extends Runnable {

    /**
     * This method is called by the Ncs Java vm before the
     * thread is started.
     */
    public void init();

    /**
     * This method is called by the Ncs Java vm when the thread
     * should be stopped. Stopping the thread is the responsibility of
     * this method.
     */
    public void finish();

}

An example of an application component implementation is found in SNMP Notification Receiver.

User Implementations typically needs resources like Maapi, Maapi Transaction, Cdb, Cdb Session etc. to fulfill their tasks. These resources can be instantiated and used directly in the user code. Which implies that the user code needs to handle connection and close of additional socket used by these resources. There is however another recommended alternative, and that is to use the Resource manager. The Resource manager is capable of injecting these resources into the user code. The principle is that the programmer will annotate the field that should refer to the resource rather than instantiate it.

@Resource(type=ResourceType.MAAPI, scope=Scope.INSTANCE)
public Maapi m;

This way the NSO Java VM and the Resource manager can keep control over used resources and also have the possibility to intervene e.g. close sockets at forced shutdowns.

The Resource manager can handle two types of resources, Maapi and Cdb.

package com.tailf.ncs.annotations;

/**
 * ResourceType set by the Ncs ResourceManager
 */
public enum ResourceType {

    MAAPI(1),
    CDB(2);
}

For both the Maapi and Cdb resource types a socket connection is opened towards NSO by the Resource manager. At a stop the Resource manager will disconnects these sockets before ending the program. User programs can also tell the resource manager when its resources are no longer needed with a call to ResourceManager.unregisterResources().

The resource annotation has three attributes:

package com.tailf.ncs.annotations;

/**
 * Annotation class for Action Callbacks Attributes are callPoint and callType
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface Resource {

    public ResourceType type();

    public Scope scope();

    public String qualifier() default "DEFAULT";

}

package com.tailf.ncs.annotations;

/**
 * Scope for resources managed by the Resource Manager
 */
public enum Scope {

    /**
     * Context scope implies that the resource is
     * shared for all fields having the same qualifier in any class.
     * The resource is shared also between components in the package.
     * However sharing scope is confined to the package i.e sharing cannot
     * be extended between packages.
     * If the qualifier is not given it becomes "DEFAULT"
     */
    CONTEXT(1),
    /**
     * Instance scope implies that all instances will
     * get new resource instances. If the instance needs
     * several resources of the same type they need to have
     * separate qualifiers.
     */
    INSTANCE(2);
}

When the NSO Java VM starts it will receive component classes to load from NSO. Note, that the component classes are the classes that are referred to in the package-meta-data.xml file. For each component class the Resource Manager will scan for annotations and inject resources as specified.

However the package jars can contain lots of classes in addition to the component classes. These will be loaded at runtime and will be unknown by the NSO Java VM and therefore not handled automatically by the Resource Manager. These classes can also use resource injection but needs a specific call to the Resource Manager for the mechanism to take effect. Before the resources are used for the first time the resource should be used, a call of ResourceManager.registerResources(...) will force injection of the resources. If the same class is registered several times the Resource manager will detect this and avoid multiple resource injections.

MyClass myclass = new MyClass();
try {
    ResourceManager.registerResources(myclass);
} catch (Exception e) {
    LOGGER.error("Error injecting Resources", e);
}

The AlarmSourceCentral and AlarmSinkCentral that is part of the NSO Alarm API can be used to simplify reading and writing alarms. The NSO Java VM will start these centrals at initialization. User implementations can therefore expect this to be set up without having to handle start and stop of either the AlarmSinkCentral or the AlarmSourceCentral. For more information on the alarm API see Alarm Manager in User Guide

As stated above the NSO Java VM is executed in a thread implemented by the NcsMain. This implies that somewhere a java main() must be implemented that launches this thread. For NSO this is provided by the NcsJVMLauncher class. In addition to this there is a script named ncs-start-java-vm that starts java with the NcsJVMLauncher.main(). This is the recommended way of launching the NSO Java VM and how it is set up in a default installation.

If there is a need to run the NSO Java VM as an embedded thread inside another program. This can be done as simply as instantiating the class NcsMain and start this instance in a new thread.

NcsMain ncsMain   = NcsMain.getInstance(host);
Thread  ncsThread = new Thread(ncsMain);

ncsThread.start();

However, with the embedding of the NSO Java VM comes the responsibility to manage the life-cycle of the NSO Java VM thread. This thread cannot be started before NSO has started and is running or else the NSO Java VM control socket connection will fail. Also, running NSO without the NSO Java VM being launched will render runtime errors as soon as NSO needs NSO Java VM functionality.

To be able to control an embedded NSO Java VM from another supervising java thread or program an optional JMX interface is provided. The main functionality in this interface is listing, starting and stopping the NSO Java VM and its Component Managers.

NSO has extensive logging functionality. Log settings are typically very different for a production system compared to a development system. Furthermore, the logging of the NSO daemon and the NSO Java VM is controlled by different mechanisms. During development, we typically want to turn on the developer-log. The sample ncs.conf that comes with the NSO release has log settings suitable for development, while the ncs.conf created by a "system install" are suitable for production deployment.

The NSO Java VM uses Log4j for logging and will read its default log settings from a provided log4j2.xml file in the ncs.jar. Following that, NSO itself has java-vm log settings that are directly controllable from the NSO CLI. We can do:

admin@ncs(config)# java-vm java-logging logger com.tailf.maapi level level-trace
admin@ncs(config-logger-com.tailf.maapi)# commit
Commit complete.

This will dynamically reconfigure the log level for package com.tailf.maapi to be at the level trace. Where the java logs actually end up is controlled by the log4j2.xml file. By default the NSO Java VM writes to stdout. If the NSO Java VM is started by NSO, as controlled by the ncs.conf parameter /java-vm/auto-start, NSO will pick up the stdout of the service manager and write it to:

admin@ncs(config)# show full-configuration java-vm stdout-capture
java-vm stdout-capture file /var/log/ncs/ncs-java-vm.log

(The "details" pipe command also displays default values)

The section /ncs-config/japi in ncs.conf contains a number of very important timeouts. See $NCS_DIR/src/ncs/ncs_config/tailf-ncs-config.yang and ncs.conf(5) in Manual Pages for details.

Whenever any of these timeouts trigger, NSO will close the sockets from NSO to the NSO Java VM. The NSO Java VM will detect the socket close and exit. If NSO is configured to start (and restart) the NSO Java VM, the NSO Java VM will be automatically restarted. If the NSO Java VM is started by some external entity, if it runs within an application server, it is up to that entity to restart NSO Java VM.

When using the auto-start feature (the default), NSO will start the NSO Java VM as outlined in the section called “Overview”, there are a number of different settings in the java-vm YANG model (see $NCS_DIR/src/ncs/yang/tailf-ncs-java-vm.yang) that control what happens when something goes wrong during the startup.

The two timeout configurations connect-time and initialization-time are most relevant during startup. If the Java VM fails during the initial stages (during INIT_JVM, LOAD_SHARED_JARS, or LOAD_PACKAGE) either because of a timeout or because of a crash, NSO will log "The NCS Java VM synchronization failed" in ncs.log.

After logging NSO will take action based on the synchronization-timeout-action setting:

If you have problems with the Java VM crashing during startup, a common pitfall is running out of memory (either total memory on the machine, or heap in the JVM). If you have a lot of Java code (or a loaded system) perhaps the Java VM did not start in time. Try to determine the root cause, check ncs.log and ncs-java-vm.log, and if needed increase the timeout.

For complex problems, for example with the class loader, try logging the internals of the startup:

admin@ncs(config)# java-vm java-logging logger com.tailf.ncs level level-all
admin@ncs(config-logger-com.tailf.maapi)# commit
Commit complete.

Setting this will result in a lot more detailed information in ncs-java-vm.log during startup.

When the auto-restart setting is true (the default) it means that NSO will try to restart the Java VM when it fails (at any point in time, not just during startup). NSO will at most try three restarts within 30 seconds, i.e. if the Java VM crashes more than three times within 30 seconds NSO gives up. You can check the status of the Java VM using the java-vm YANG model. For example in the CLI:

admin@ncs# show java-vm
java-vm start-status started
java-vm status running

The start-status can have the following values:

The status can have the following values: