Application Development Concepts

This page will target different concepts that are generally applicable for all IOx applications.

Application States

IOx framework provides a consistent mechanism for application lifecycle management on IOx devices.

An application residing on the Fog node, will be in any of the following states:

• DEPLOYED: Application is installed on the device. Resources needed by the application is not committed to the application.
• ACTIVATED: The resources required by the application is now committed. Associated container artifacts are also generated.
• RUNNING: Application is now running
• STOPPED : Application is stopped.

The below state diagram summarizes various actions that can be performed on an application and transition rules.

NOTE: deploy is same as install undeploy is same as uninstall Some of the operations such as backup_data, download_data, restore_data etc., are not available yet.

The administrative tools may choose to handle few states internally to provide better user experience. For example, Fog Director doesn't have explicit ACTIVATE and DEACTIVATE flows and are done under the hood.

What is Activation and Deactivation?

Applications asks are described in package.yaml. For example, the application may need one or more serial ports, it may need networking where certain ports are needed by the application.

At development time, a developer will only be able to articulate such requirements. However, how those requirements are made available to the application can only be decided at deployment time by the administrator.

In the above example, the exact serial port that is connected with sensors will need to be made available to the application. The interface requested by the application may be connected to a specific logical network on the device as decided by the administrator.

Further, the administrator may need to change the resource provisioning. For instance, the serial port where there sensors were connected had to be changed to a different one. The app may need to be scaled up by granting more cpu/memory resources.

In order to facilitate the ability to generically state requirements but enable the association or provisioning of those requirements at a later point in time, the activation concept is introduced.

After deployment, the application is available on the device. The administrator can then provision the right resources during activation time at which point the resources are committed to the application. If the resource allocation or mapping needs to change, the administrator can DEACTIVATE and ACTIVATE again with the updated mapping without having to uninstall and reinstall the application again.

Requesting resources

Applications must describe its resource requirements in package.yaml (descriptor) file.

Refer package descriptor documentation for more details about available options and schema rules.

We will look at a sample descriptor file and go through different sections.

Copydescriptor-schema-version: "2.0"

info:
  name: PySBOT
  description: "Writes collected sensor data to log files"
  version: "1.5"
  author-link: "http://www.cisco.com"
  author-name: "Cisco Systems"

app:
  # Indicate app type (vm, paas, lxc etc.,)
  type: paas

  resources:
    profile: c1.small

    network:
      -
        interface-name: eth0
        ports:
            tcp: [6000]
            udp: [9000,1000]
    devices:
      -
        type: serial
        label: HOST_DEV1
        usage: To be connected to gps sensor

  # Specify runtime and startup
  startup:
    runtime: python
    runtime-version: 2.7.3
    target: main.py

• It is indicated that the application is a "paas" style application.
• The app is requesting a resource profile called c1.small. The definition of each resource profile is published and maps a certain CPU units(100) and memory (32 MB).
• Network requirements
• There is a provision for app to request for multiple network interfaces.
• NOTE: Currently only a single interface is supported. And the interface-name HAS to be eth0
• Multiple TCP and UDP ports can be requested
• At the time of activation, the administrator has to associate eth0 to a specific logical network (ex. iox-nat0)
• Device requirements
• An app can request for multiple devices depending on the target IOx platform.
• In this example, the app is a requesting for a single device of type "serial".
• The usage can be used a hint for the administrator to indicate how the app would use the serial port
• The label field is used to logically identify the requirement
  • The administrator will use the value in the label field., i,e HOST_DEV1 and associate it with the physical serial port (ex. /dev/ttyS1)
  • IOx framework sets up an environment variable HOST_DEV1: /dev/ttyS1 and makes it available to the application.
  • This enables the application code to be written generically without knowing the actual serial device id at development time. Below is a snippet that is querying the HOST_DEV1 environment variable to determine the serial device ID before connecting to it.

Copy        serial_dev = os.getenv("HOST_DEV1")
        sdev = serial.Serial(port=serial_dev, baudrate=9600)

Scaling to multi cores

On platforms that provide multi core compute, applications can leverage those resources.

For any container based applications (PaaS, LXC< Docker), all cores on the host will be exposed to the containers. The applications can detect the number of cores on the system programmatically and tune themselves to leverage available resources. Detecting number of cores on the system is language specific. For ex., in python you could do:

Copyimport multiprocessing

multiprocessing.cpu_count()

For VM style applications, the developer can ask for number of vcpus required by the application. These VCPUs are essentially virtual cpu threads presented to the VM by qemu/kvm subsystem. Further, it is even possible to customize the cpu topology. Below is a snippet showing the resources section requiring vcpus.

Note: Specifying VCPUs and cpu topology are only available from schema version 2.2 onwards

Copydescriptor-schema-version: "2.2"
info:
  name: Windows 7 VM
  description: "Sample Windows 7 VM"
  version: "0.1"
  author-link: "http://www.cisco.com/"
  author-name: "Cisco Systems"

app:
  type: vm
  cpuarch: "x86_64"
  resources:
    profile: custom
    cpu: "600"
    disk: "10"
    # 3G memory
    memory: "3072"
    vcpu: "4"
    cpu-topology:
      sockets-per-core: "2"
      cores: "2"

...
...

• The app is asking for a total of 4 vcpus.
• The cpu topology also specifies how this has to be presented to the application.
• Note: If cpu-topology is specified, sockets-per-core * cores == vcpu

Application Health Monitoring

IOX allows to run third party applications on the devices. Though these applications uses IOX infrastructure for app life cycle management. IOX has a very little visibility on how the application is performing.

CAF currently only monitors the container, inside which app is running. It assumes if container is up and running it means application is performing well. This may not be true in all the cases.

We have seen often customers asking that "my container is up but my application is not running"

CAF app monitoring support will address specifically this issue. Below is the snippet which shows how to package the app health monitoring script. This monitoring script will monitor the health of the application. It will be executed periodically by CAF monitoring service. Return Status of 0 indicates SUCCESS. Non zero return code will be regarded as failure.

Copyapp:
  # Indicate app type (vm, paas, lxc etc.,)
  cpuarch: "x86_64"
  type: lxc 
  kernel-version: "3.16.0"

  resources:
    profile: c1.small
 
    network:
      - 
        interface-name: eth0
        description: Management
        ports: 
            tcp: ["9000-9999"]
#            udp: [10000]
  monitor:
    script: /bin/healthprobe.sh
    initial_delay_seconds: 30
    period_seconds: 60

...
...

• Application has specified /bin/healthprobe.sh as app health script. This will be executed periodically.
• initial_delay_seconds specifies wait for specified seconds after the application starts before executing the health monitoring script. Default Value 30
• period_seconds Periodicity in seconds when this scripts needs to run. (multiple of 30 seconds) Default value 60.

Environment Variables

CAF provisions a set of environment variables for applications.

Also, for PaaS style applications, additional variables are made available:

Further to the above variables, CAF will also provision additional variables depending on the application.

Accessing environment variables

Depending on the application type, accessing the environment variables may differ.

• PaaS style applications: Since CAF manages the startup of the application, the environment variables are injected into the process space of the application. So, the app will be able to access using the equivalent of getenv method. For example, a python app can use os.getenv("CAF_PERSISTENT_DIR") to get the value of persistent directory provisioned for the app.
• VM and LXC style applications: Since the applications control their startup sequence, CAF has no reliable way of injecting the variables into the VM/LXC container space. Therefore, the environment variables are written to /data/.env file in the context of the container. The content of this file is readily sourceable and the application can then make use of the env variables.
• Docker applications: Environment variables are injected into the process space of the application. In addition, the environment variables are written to the /$CAF_APP_PERSISTENT_DIR/.env file in the context of the container. The content of this file is readily sourceable and the application can then make use of the env variables.

Ex: Contents of /data/.env file:

Copy#!/bin/sh
export CAF_APP_LOG_DIR=/data/logs
export CAF_APP_CONFIG_FILE=/data/package_config.ini
export CAF_APP_PERSISTENT_DIR=/data
export CAF_APP_CONFIG_DIR=/data
export CAF_APP_USERNAME=root

Bootstrap configuration file

Applications may want to externalize certain variables whose values will need to be configurable at the time of deployment or can be updated while installed on the device.

CAF enables this via bootstrap configuration file. This file should be named package_config.ini and should be present in the root of the application package. Administration tools (Fog Director, Local Manager, ioxclient) provide ability to modify this file so that the values can be customized to a deployment environment.

For instance, if an application wants to know it's upstream server's IP/Port to connect and push data, it is ideal that such configurable entities is externalized into the bootstrap config file.

Sample package_config.ini file:

Copy[upstream]
server_ip: 72.111.231.21
server_port: 7623

Note: The content of this file should be valid .ini format

This file can be modified with right values using the admin tools. The application itself can access this file by reading the value pointed by environment variable CAF_APP_CONFIG_FILE,

Safeguarding against flash wear

Many platforms have flash storage as the primary mode of storage:

• Flash storage has limited P/E (program/erase) cycles and is susceptible to flash wear
• It is OK to use flash for one-time or infrequent cases like saving startup configuration, images, etc. When used for only these cases, the flash tends to last the lifetime of the platform.
• It is not OK to use flash for dumping verbose logs and frequently changing data which can drastically reduce the lifetime of the flash
• To safeguard against this, the developer should be aware of size, scope and frequency of the content being written on the flash when the application is running and minimize it as much as feasible

Following are some recommended ways of minimizing flash wear:

• Reduce frequency of writing on flash
• Use compact log formats
• Limit size of log files to be written on the platform
• Use circular logs to keep the recent ones, overwriting older logs
• Avoid storing application data which is not needed for application state persistence

NOTE: To avoid unpredicatable flash wear that leads to data corruption, certain platforms prohibit the use of bootflash for application hosting. Consult the "Plastforms" section for the specific platform limitations.

Application logging and persistent storage

Here are a few things to keep in mind while writing log files from your application.

• To restrict flash wear, it is advised that applications write only the essential information (any data, log files etc.,) to persistent storage in production.
• If your application log files have to be manageable via IOx and persisted, always write to the directory indicated by the environment variable CAF_APP_LOG_DIR
• For any data that needs to be persisted (database files, configuration files etc.,) always write them to the directory indicated by the environment variable CAF_APP_PERSISTENT_DIR
• If applications write to STDOUT or STDERR, it will be visible on the application's console. This will make connecting and issuing commands in the application console during debugging cumbersome. Therefore, it is advised that normal log messages MAY NOT be written to STDOUT.

Application File Management

IOx provides ability to read and update application's bootstrap configuration (package_config.ini). In addition to that, applications may need several generic files to be present for its operation which falls outside the scope of package_config.ini.

To enable this, CAF provides generic file management functionality. Each application is provisioned with an environment variable CAF_APP_APPDATA_DIR which points to a location in the application's root file system. Apps can read or write files from this location. The files under this location can be managed via CAF REST APIs and admin tools such as ioxclient, Local Manager etc., provide mechanisms to do it.

Here is a sample ioxclient command:

Copyioxclient app appdata upload <app_id> <input_file> <target_path>
e.g
 ioxclient app appdata upload lxc ./activation.json  activation.json

The above command will upload activation.json under app data directory specified by env variable CAF_APP_APPDATA_DIR.

Application can access this as $CAF_APP_APPDATA_DIR/activation.json

Accessing Host File system

Application may require to need to access the host file system. Currently applications have only access to /data which is provisioned for the application.

Constraints/Assumptions

• Each platform will expose only set of directories/files or mount points that application can ask.
• The application does not have the exclusive access on the provisioned mount point and it shares it with the host which also have the access to the same folder.
• In case of writable mounts, application is the owner of the files/directories inside its folder, other application will not be able to access them.
• There is no limit on the size of the files/folders created by an app on the provisioned mount point in app.

For getting the list of host directories/files which can be shared on the application:

Copyioxclient platform cap

 ...
   "supported_host_mount_paths": [
    "/software/caf/app_data_share",
    "/var/log/app_tmpfs"
   ],

Package.yaml changes for requesting host file system access

Application can specify the need of using the host file system in package.yaml. It can ask multiple host file system access point if the platform supports more than one. It will provisioned by the admin similar to the device requirements of the application.

Application needs to specify the host paths in host_mounts section of package.yaml for accessing the host file system. Inside this section it can specify host paths that app wants to access and the description of the purpose for which it will be used so admin can take a informed choice of provisioning the host file system to the app.

Copyresources:
    profile: c1.small

    host_mounts:
      -
        target_mount: "/mnt/store_fwd/"
        description: "used for store and forward"
      -
        target_mount: "/tmp/var/tmpfs"
        description: "used for tmpfs"

During activation of this app, admin will provide required mappings for the host file system access. Once the application is successfully activated. It can use mount points /mnt/store_fwd and /tmp/var/tmpfs for accessng the host file system paths.

Cartridges

A cartridge is a deployable, pluggable piece of software that provides functionality and content to be consumed by the requesting application or service on an IOx node.

An application can request for cartridges by specifying the requirements in the descriptor file (package.yaml).

For example, when an application requests for a particular language runtime :

Copy  startup:
    runtime: python
    runtime-version: 2.7.3
    target: main.py

This requirement of python 2.7.3 is met by means of a cartridge. CAF identifies this requirement and finds out if a relevant cartridge providing python 2.7.3 functionality is available on the platform. Fog director also has the ability to push required cartridges on demand to a device before installing the application. If a matching cartridge is found, CAF provisions the cartridge into the application's runtime environment. In this example, that makes python 2.7.3 available to the application.

Handling signals

CAF will send a SIGTERM signal to your application when a "stop" event is triggered via its API. The application is expected to register appropriate signal handlers and shutdown gracefully.

If the application doesn't shut itself down in 10 seconds, CAF will send a SIGKILL at which time the application will be ungracefully terminated.

Therefore, your application must handle at least SIGTERM so that a graceful stop is possible.

For example, if you are using Python, you can do something like:

Copyimport signal

def _sleep_handler(signum, frame):
    print "SIGINT Received. Stopping CAF"
    stop_my_app()

def _stop_handler(signum, frame):
    print "SIGTERM Received. Stopping CAF"
    stop_my_app()

signal.signal(signal.SIGTERM, _stop_handler)
signal.signal(signal.SIGINT, _sleep_handler)

A note about platform/CAF powering down

All applications are launched and managed by CAF. So, if CAF is shutdown (which typically only happens during maintenance, or the platform is powered down), CAF will stop and deactivate all the RUNNING applications. The previous state of the applications will be restored when CAF is restarted.

Application core files

Applications can crash or behave abnormally. For analysis and debugging in such cases, access to application's core dumps may be required. CAF provides the ability to view and download core files generated by applications.

Refer to ioxclient application cores CLIs and its usage

To enable core file dumping during debugging, connect to application's console and set:

Copyulimit -c unlimited

PaaS application concepts

Activating with "Debug" mode

When a paas application is deployed in IOx, the hosting framework composes a container, provisions the application requirements and runs the application within the container.

Since the application is the primary reason the container exists, if the main application process (as specified in startup->target directive in package.yaml) dies, CAF also brings down the container and the application is marked as stopped.

Some examples to make this clear:

• Let us consider a simple application that just comes up and prints "Hello World!". This means that the main application process terminates immediately after the print is finished. CAF will detect this condition and stops the application container automatically.

• If the application comes up and is doing some operations in a loop, but crashes due to errors, CAF will detect this condition and stops the application container automatically.

This brings up a question - "How do I debug the root cause of my application crash within the container context?". Because, as seen above, if the main application process terminates for any reason, CAF automatically brings the container down.

To facilitate this need for debugging, IOx supports application activation in debug mode.

Here is an example of how to enable this using ioxclient. The activate command has a --debug flag.

Copy~ ❯❯❯ ioxclient application activate
NAME:
   activate - Activate application

USAGE:
   command activate [command options] <application_id>

OPTIONS:
   --payload 	Pass the path to a JSON file containing your payload
   --debug 	Set to on/off.

Set it to on, to use debug mode.

Copy~ ❯❯❯ ioxclient application activate --debug on nt
Currently using profile :  local
Command Name: application-activate
App nt is Activated

When the application is activated with debug mode, even if the main application process crashes, the container is kept running. The developer can then console into the application for further debugging.