1. Creating a simple application (basic)

In the following sections, we'll go through steps to build a simple package and deploy it to the blade.

Build your application

As mentioned in the requirement section, the 3rd party development should be done on a linux machine and should be compatible with the kernel that AXP provides. Third party development can be done in Java, C/C++, shell scripts, Python, and Perl. Building your application should not be any different than building any other linux application. Just make sure you include all the libraries required when packaging your application.

For this tutorial, we'll use a simple Hello app, which is a simple bash script. We'll show you how to package this app and get it installed onto the blade.

#!/bin/bash echo "Hello...##########################################"

APPLICATIONEXTENSIONPLATFORM:Return To Top

Package the application

Extract the SDK

You'll first need to get the packaging tool:

• Locate the sdk package in your "aesop-root/all-images" directory for the "sdk.tar.gz" file. You will see this directory in your workspace root.
• Extract the tarball into any directory on your machine "tar -xzf sdk.tar.gz".

Note: if you do not see the tarball under aesop-root, then either your workspace has not been build, or "make install" has not been done to install the image into this directory. Do a "make" and "make install" in the workspace root.

Create a Root Directory

Create a common root directory. This directory will be the root directory from which the packages will be created and then installed onto the AXP platform. Copy all relevent files and links including their directories into the common root directory. The root directory I have created for my app is in /users/home/jescheng/source.

Everything under this directory will appear in the root directory of my vserver with the directory structure intact.

Note: If you're accessing the blade as the host, you'll see your application's root directory in /home/vserver/[APPLICATIONEXTENSIONPLATFORM:AppName]/

Create a Project Directory

Create a project directory for the packaging tool. You'll find your resulting package in this directory. The project directory I used for my app is /ws2/jescheng/project/

Packaging information

You'll need to prepare some information before packaging your application. Here is a complete list of information and some description of each. Note that not all of them are required, some are labeled with optional. The information here is provided for completeness (straight from developer guide), you can skip over to the next section if you just want to package with basic options.

• Project Directory¿Directory that stores generated package files, templates, and temporary files created during packaging.
• Development Certificate location ¿X.509 certificate authorized for software development on the platform. Example: "/xyz_source/x509.cer"
• Software Development Authorization file location¿Binary file that grants permission to the certificate holder to develop software for the platform. Example: "/xyz_source/dev_auth"
• Private Key location¿File containing the RSA key that is used to sign the application vendor software. Example: "/xyz_source/privkey.txt"
• Application Name¿Name of the packaged application. Example: "XYZ"
• Application Version¿Version of the packaged application. Example: "1.0"
• Application Description¿(optional) Description of application functionality. For example:"XYZ Utility"
• Application UUID¿(optional) Application unique identifier. It is automatically generated when not specified.
• Application Files Location¿Directory that contains application files to be packaged. The directory structure should mirror the desired directory structure on the target file system. Example: "/xyz_source/root_fs"
• List of files to be protected¿(optional) Plain text document that contains a list of files (one file per line) that will be checked for tampering before application initialization.
• Platform License¿ Specifies modern POSIX regular expression for target platforms where this software is allowed to run. The application is only installed and allowed to run if the target platform matches the specified regular expression.
• Package Dependencies¿Select from a list of packages in the SDK and project directories. The user can also specify UUID and filename for a package that exists on the development workstation file system. After each dependency, the user has to specify a dependency type:
  o - From [APPLICATIONEXTENSIONPLATFORM:version]¿Dependency must be of the specified or later version.
  o - To [APPLICATIONEXTENSIONPLATFORM:version]¿Dependency must be of the specified or earlier version.
  o - Exclude [APPLICATIONEXTENSIONPLATFORM:version]¿ Package cannot be installed with other dependencies for the specified version.
  o - Only [APPLICATIONEXTENSIONPLATFORM:version]¿Dependency must be of the specified version.
  o - None¿Package cannot be installed with the specified dependency.
  o - All¿Dependency of any version must be present for the package installation.
• Disk Limit (in megabytes) - Set application disk usage limit. Disk usage includes all application binary and data files as well as any diagnostics data generated by the customer when using the application (such as log and core files). Target platform resources must be taken into consideration when selecting a value for this option. Application may fail to install or run if the requested resource is not available on the target platform. Example: 1500Mb
• Memory Limit (in megabytes) - Set application RAM usage limit. Target platform resources must be taken into consideration when selecting a value for this option. Application may fail to install or run if requested resource is not available on target platform. Example: 256Mb
• CPU Limit (in CPU index points) - Sets the maximum CPU utilization that an application can access. CPU index will be calculated using a benchmark and published for each supported hardware platform. A more powerful platform will have a higher CPU index and will be able to accommodate a higher number of concurrent applications. Target platform resources must be taken into consideration when selecting a value for this option. Application may fail to install or run if enough CPU resources are not available on the target platform. Example: 50Mb
• Application capacity¿ Textual description of application capabilities. Example: This string may indicate the call processing rate for voice routing application.
• Post-install script ¿(Optional) Specifies the script to run after installation. Example: /bin/postinstall.sh

Run the packaging tool

Look into the sdk directory you just created, under "tools" there should be a script called pkg_build.sh. You can run this script in interactive or non-interactive mode.

Interactive Mode

Following is a sample output when packaging the Hello app, with the interactive mode (redundant output such as description has been removed from the output to reduce size).

[APPLICATIONEXTENSIONPLATFORM:jescheng@jescheng-lnx /ws2/jescheng/sdk/tools]$ pkg_build.sh SLIM Packaging
Utility. (C) 2007 Cisco Systems, Inc Checking dependencies... No command
parameters specified - entering interactive mode. ** project-dir Project
Directory: /ws2/jescheng/project ** dev-cert Development Certificate
File: /vws/vld/devtest-pkg/devcert/dev_certificate.jvln ** dev-auth
Development Authorization File: /vws/vld/devtest-pkg/devcert
/dev_authorization.jvln ** private-key Private Key for Signing Operations:
/vws/vld/devtest-pkg/devcert/private.key ** name Application Name: Hello

• • version Version: 1 ** description (optional) Description: echo hello
  • uuid (optional) Unique Identifier: Using Default Value: ** source-dir
    Source Directory: /users/jescheng/source ** protect (optional) Protect List
    File: Using Default Value: ** deps (optional) Dependency Subsystem
    Identifier: Using Default Value: ** disk-limit resource limit parameters are
    'min' and 'opt' Enter resource limit parameter: min Minimum Limit: 50M
    resource limit parameters are 'min' and 'opt' Enter resource limit parameter:
    opt Optimal Limit: 50M query_disk_limits min=50M,opt=50M ** memory-limit
    Memory Limit: 50M ** cpu-limit resource limit parameters are 'min' and 'opt'
    Minimum Limit: 1 resource limit parameters are 'min' and 'opt' Enter resource
    limit parameter: opt Optimal Limit: 1 query_cpu_limits min=1,opt=1
  • postinstall (optional) Post-install script path: Using Default Value:
    Interactive mode complete.

Non-Interactive Mode

We can do the same work without the interactive mode. The command below will produce the same package.

./pkg_build.sh --project-dir '/ws2/jescheng/project' --dev-cert
'/vws/vld/devtest-pkg/devcert/dev_certificate.jvln' --dev-auth
'/vws/vld/devtest-pkg/devcert/dev_authorization.jvln' --private-key
'/vws/vld/devtest-pkg/devcert/private.key' -name 'Hello'
--version '1' --description 'echo hello' --source-dir
'/users/jescheng/source' --disk-limit 'min=50M,opt=50M' --memory-limit
'50M'--cpu-limit 'min=1,opt=1'

APPLICATIONEXTENSIONPLATFORM:Return To Top

Retrieve Package

After the packaging tool completes, you should find the package in the project directory you specified under "pkg". There should be a .pkg and a .prt1 file. Copy them to your ftp directory.

[APPLICATIONEXTENSIONPLATFORM:jescheng@jescheng-lnx /ws2/jescheng/project/pkg]$ ls Hello.1.pkg
Hello.1.prt1 [APPLICATIONEXTENSIONPLATFORM:jescheng@jescheng-lnx /ws2/jescheng/project/pkg]$cp
Hello.1.* /var/ftp/

APPLICATIONEXTENSIONPLATFORM:Return To Top

Install an Application Package to the Blade

Install the packages using the software install add url url command where url is the ftp location of the package.

se-1-100-50-222.unspecified> $software install add url
ftp://128.107.146.24/Hello.1.pkg WARNING:: This command will install the
necessary software to WARNING:: complete an add-on install. It is
recommended that a backup be done WARNING:: before installing software.
Would you like to continue? [n] y Downloading ftp Hello.1.pkg Bytes
downloaded : 4745 ....

Note: Both .pkg and pkg.prt1 files must be present in the ftp server.

APPLICATIONEXTENSIONPLATFORM:Return To Top

Verify installation

You can verify the state of your application by executing the command:

se-1-100-50-222.unspecified> sh app-service state APPLICATION
STATE HEALTH Hello online

APPLICATIONEXTENSIONPLATFORM:Return To Top

Start your application

There are two ways to start your application: Manually or Automatically.

Please refer to VSEapp.AXP3rdPartyApplicationsDevelopmentFAQ for more information.

2. Application development

a. Handling RPM files

Extracting RPM files with standard rpm, rpm2cpio commands (phase I)

Since rpm is not supported, developers who want to install RPMs as part of their application needs to extract the RPMs in their development environment and integrate it together with the rest of their application.

To extract an RPM to current directoy as if that is root

rpm2cpio /tmp/myapp.rpm | cpio -ivd

Developers can examine any pre/post install/uninstall scripts that the RPM contains

rpm -qp --scripts /tmp/myapp.rpm

Developers can examine any dependencies (libraries, other RPMs) that the RPM requires

rpm -qp -R /tmp/myapp.rpm

Extracting RPM files with RPM extractor tool (phase II)

Given that their development environments have rpm and rpm2cpio supported, there is a tool provided by the SDK to help developers quickly extract all the RPMs into the project source root directory, examine any dependencies the RPMs required, and view any pre/post install/uninstall scripts that the RPMs contains.

This tool is provided as /tools/rpm_extractor.sh inside the SDK package.

Using the RPM extractor tool

Before using this tool, developers should have a project directory created, this directory could be the same directory where packaging tool is run.

workstation# rpm_extractor.sh --proj <Project Directory> [APPLICATIONEXTENSIONPLATFORM:\-\-output]
[APPLICATIONEXTENSIONPLATFORM:\-\-scripts] [APPLICATIONEXTENSIONPLATFORM:\-\-deps] <RPM files>

Required Arguments

• --proj <Project Directory>: Designates where the project directory is located. This tool will create a sub-directory structure as under the project directory as follows:

<Project Directory>/ rpm_extractor/ output/ scripts/ deps/

Extracted RPM files will be placed in the output/ directory as if the directory is root directory. Scripts information will be placed in the scripts/ directory, the name of the file will be the RPM filename suffixed with ".scripts". Dependencies information will be placed in the deps/ directory, the name of the file will be the RPM filename suffixed with ".deps". If none of the optional arguments are specified, all three operations will be performed.

• <RPM files>: A list of RPM files. This tool can process multiple RPM files at the same time. RPM files will be extracted to the same output directory as that directory is treated as root .

Optional Arguments

• --output: Specify the extract operation to be performed
• --scripts: Specify the script extract operation to be performed
• --deps: Specify the dependency extract operation to be performed
• --help: Print a usage page

After RPM extraction

With the RPM extracted (either by using the standard rpm commands, or the rpm extraction tool), and given the scripts and dependencies information known, developers should port the files following the steps below

1. Look at any library and RPM dependencies, gather any new RPM files needed and perform the extraction
2. Examine the extracted RPM files, move them to the project source directory
3. Examine the pre/post install/uninstall scripts, write the appropiate scripts in the project source directory
4. Package the application and install it on the blade

Troubleshooting an application with extracted RPMs

It might not work the first time an application containing RPMs are installed to the blade. Typical problems include:

1. Missing libraries
If there are further libraries missing, try transfer those libraries using ftp into the blade. Developers usually don't need to repackage to test
2. Missing configurations
Sometimes the configuration scripts extracted from the RPMs might not run successfully in the first trial. Developers can access the linux shell of the application and manually invoke the scripts to see if there are any problems.

APPLICATIONEXTENSIONPLATFORM:Return To Top

b. Accessing the shell of the VServer context

Phase I

To access shell in vserver context, application developers can install their application with dependency on the application debug package. This package provides a CLI to access the linux shell within the vserver context. The steps are:

1. Package the application with dependency on the app_debug.pkg
2. After installation of the application with the app_debug.pkg, use this CLI:

blade# app-service system myapp (myapp)# linux shell bash#

Phase II

In addition to the CLI to access linux shell, application debug package provides an SSH server that runs on the vserver context in phase 2.

Vserver SSH server support is provided at two levels:

1. Production Environment

• Provides a "tunnel_user" SSH user that does not have shell access upon SSH login. Third party developers have control what this "tunnel_user" can do by writing a startup script that is called upon successful SSH login
  2. Debugging Environment
• Provides a "tunnel_user" SSH user, same as Production Environment
• Provides a "tunnel_root" SSH user that has shell access of the vserver context upon SSH login

This two levels of support is provided through two separate add-on package. Third party application that wants this support must specify either of these add-on packages as dependency upon packaging time.

1. app_ssh.pkg - addon package for Production Environment SSH support
2. app_debug.pkg - addon package for Debugging Environment SSH support

Setup vserver SSH server

Steps to setup a vserver to run SSH server:

1. Package the application with dependency on the application debugging
2. After installation of the application with the app_debug.pkg, make sure that there is an interface binded to the vserver, so that SSH connection can be made into the vserver. If not, follow these steps:

blade# config terminal (config)# app-service system myapp (config-myapp)#
bind ip eth0 (config-myapp)# end

3. Activate the "tunnel_root" user (same applies to "tunnel_user" user) by setting a password

blade# config terminal (config)# app-service system myapp (config-myapp)#
ip ssh uername tunnel_root password hello (config-myapp)# end

4. Enable the vserver SSH server

blade# config terminal (config)# app-service system myapp (config-myapp)#
ip ssh server (config-myapp)# end

5. Users can check that vserver SSH server is running

blade# app-service system myapp (myapp)# show ssh-server Application SSH
Server Status: RUNNING

6. Users can now connect to the vserver SSH server. By default, vserver SSH server port is 2022 (There is a CLI to change the port). For connection using "tunnel_root" as the SSH user, there will be shell access

workstation-shell# ssh tunnel_root@myblade -p 2022 tunnel_root@myblade's
password: vserver-shell#

7. For using the "tunnel_user" as the SSH user, upon successful connection, it will first be greeted by the AXP login message. Then, the startup script third party application provides will be processed. Third party application should provide the script /usr/ssh/home/tunnel_user_app_startup.sh. For example

#!/bin/bash
echo "Welcome Tunnel User!"
echo "Please enter 1 to do a 'pwd', or 2 to do a 'ls'"
read choice
if [APPLICATIONEXTENSIONPLATFORM: $choice -eq 1 ]; then
pwd
else ls
fi

Upon login, this is what "tunnel_user" will encounter

workstation-shell# ssh tunnel_user@myblade -p 2022
tunnel_user@myblade's password:
==Start of message from the Cisco AppRE Application SSH Support==
Tunnel User (tunnel_user) has logged in successfully
Application specific Tunnel User startup script will be
invoked (if exists)
==End of message from the Cisco AppRE Application SSH Support==
Welcome Tunnel User!
Please enter 1 to do a 'pwd', or 2 to do a 'ls'
2
tunnel_user_app_startup.sh tunnel_user_startup.sh
tunnel_user_app_startup.sh.sample
Connection to myblade closed.
workstation-shell#

APPLICATIONEXTENSIONPLATFORM:Return To Top

c. Developing in the VServer

I. Synchronizing them with the developer tree on the linux blade

• Eric RsyncHowToUseIt? II. Identifying and gathering the files that have been changed in the VServer
• Eric IdentifyingChangedFiles?

APPLICATIONEXTENSIONPLATFORM:Return To Top

d. Extending the CLI

Implementing Plugin CLI

Third party developers will use the CLI Plugin tool provided in the SDK to implement Plugin CLI. The tool is located at /tools/custom_cli of the SDK directory.

In addition, developers will need to achieve the following:

1. implement plugin CLI definitions
2. implement plugin CLI actions
3. Run the plugin CLI tool with the definitions and actions
4. Make changes in the third party main application to start the plugin CLI listener
5. Package and bundle the third party application with dependency on the plugin CLI add-on package
6. Install the bundled application
7. Run the main application, issue the CLI
8. Config CLIs

Plugin CLI definitions

Developers will need to write the plugin CLI definitions as XML format. An example of plugin CLI definition file (please note that the <xml> and <DOCTYPE> header tags must also present)

-bash-3.00$ ls
actions app cli_plugin.config myapp1_cli.xml proj proj_src sdk
-bash-3.00$ cat myapp1_cli.xml

<?xml version='1.0' encoding='UTF-8' ?>

<!DOCTYPE cli_grammar SYSTEM "cli_grammar.dtd" [
<!ENTITY num '[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?'>
<!ENTITY ipaddr '[APPLICATIONEXTENSIONPLATFORM:1\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?[APPLICATIONEXTENSIONPLATFORM:0\-9]?\.[APPLICATIONEXTENSIONPLATFORM:1\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?[APPLICATIONEXTENSIONPLATFORM:0\-9]?\.
[APPLICATIONEXTENSIONPLATFORM:1\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?[APPLICATIONEXTENSIONPLATFORM:0\-9]?\.
[APPLICATIONEXTENSIONPLATFORM:1\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?[APPLICATIONEXTENSIONPLATFORM:0\-9]?'>
<!ENTITY portNum '[APPLICATIONEXTENSIONPLATFORM:1\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]*'>
<!ENTITY login '[APPLICATIONEXTENSIONPLATFORM:a\-zA\-Z]+'>
<!ENTITY password '[APPLICATIONEXTENSIONPLATFORM:a\-zA\-Z0\-9]+'>
<!ENTITY range '[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?'> ]>

<cli_grammar>
<mode>
<pattern type='id'>exec</pattern>
<cmd>
<pattern type='id'>math</pattern>
<help>Math operations with Java actions</help>
<param>
<pattern type='regex'>[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?</pattern>
<pattern type='help'>NUM</pattern>
<help>0-99</help>
</param>
<param>
<pattern type='id'>add</pattern>
<help>Add operation</help>
</param>
<param>
<pattern type='regex'>[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?</pattern>
<pattern type='help'>NUM</pattern>
<help>0-99</help>
</param>
<action paginate="yes" pipehelp="yes">
<class>exec_math.class</class>
</action>
</cmd>
<cmd>
<pattern type='id'>math</pattern>
<help>Math operations with Java actions</help>
<param>
<pattern type='regex'>[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?</pattern>
<pattern type='help'>NUM</pattern>
<help>0-99</help>
</param>
<param>
<pattern type='id'>sub</pattern>
<help>Subtract operation</help>
</param>
<param>
<pattern type='regex'>[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?</pattern>
<pattern type='help'>NUM</pattern>
<help>0-99</help>
</param>
<action paginate="yes" pipehelp="yes">
<class>exec_math.class</class>
</action>
</cmd>
</mode>
<mode>
<pattern type='id'>config</pattern>
<cmd>
<pattern type='id'>store</pattern>
<help>Store a value</help>
<param>
<pattern type='id'>key_1</pattern>
<help>Key 1</help>
</param>
<param>
<pattern type='regex'>[APPLICATIONEXTENSIONPLATFORM:0\-9][APPLICATIONEXTENSIONPLATFORM:0\-9]?</pattern>
<pattern type='help'>NUM</pattern>
<help>0-99</help>
</param>
<action>
<class>conf_store.class</class>
</action>
</cmd>
</mode>
</cli_grammar>

This will plugin 2 CLIs to the CLI console, one config CLI, another exec CLI:

• exec CLI: math <num> [ add | APPLICATIONEXTENSIONPLATFORM:sub ] <num>
• config CLI: store key_1 <num>

Plugin CLI actions

Developers should then implement the action classes referenced in the CLI definitions. For example, implementing "exec_math.java":

import java.lang.Integer;

public class exec_math{
   public static String[] eval(String tokens[], String grammar[]){
      System.out.println("exec_math.eval: called");      
      /*
      math_j  add 
      math_j  sub 
      */
      int num1 = Integer.parseInt(tokens[1]);
      int num2 = Integer.parseInt(tokens[3]);
      String operation = grammar[2];
      int result = 0;
      if (operation.equals("add")){
         result = num1 + num2;
      } else {
         result = num1 - num2;
      }
      
      String retStrArray[] = new String[2];
      retStrArray[0] = "0";
      retStrArray[1] = "exec_math.eval: answer is " + result + ".";
      return retStrArray;
   }
}

An example Makefile to compile these action classes:

JAVAC = javac
default: bundle
bundle: compile
mv *.class actions/
compile: action_dir
$(JAVAC) exec_math.java conf_store.java
action_dir:
mkdir actions
clean:
@rm -rf *~ actions

The following is an example how developers compile the actions

-bash-3.00$ ls
actions app cli_plugin.config myapp1_cli.xml proj proj_src sdk
-bash-3.00$ cd actions/
-bash-3.00$ ls
conf_store.java exec_math.java Makefile
-bash-3.00$ make
mkdir actions
javac exec_math.java conf_store.java
mv *.class actions/
-bash-3.00$ cd actions/
-bash-3.00$ ls
conf_store.class exec_math.class

Running the CLI Plugin tool

After defining the CLI definitions and implementing the CLI actions, developers should now use the CLI Plugin tool to package the CLIs into the project source directory. They need to copy the CLI definitions and CLI actions into the appropiate directories, as well as edit the cli_plugin.config file with the correct values

bash-3.00$ cd ../../sdk/tools/custom_cli/
bash-3.00$ ls
actions cli_plugin.config definitions
cli_grammar.dtd cli_plugin.py internal
bash-3.00$ cp ../../../actions/actions/* actions/
bash-3.00$ cp ../../../myapp1_cli.xml definitions/
bash-3.00$ cat cli_plugin.config
file=myapp1_cli.xml
javapath=/usr/java/java/bin
vm=myapp1
proj_src=/tmp/proj_src
bash-3.00$ python cli_plugin.py
cli_plugin.py ran successfully, please check the packages at output/
package has been bundled into the project source directory at
/tmp/proj_src

Developers may verify that certain directories (e.g. cli_comm) are created in the project source directory

-bash-3.00$ cd /tmp/proj_src/
-bash-3.00$ ls
cli_comm etc usr

Start the plugin CLI listener in the main application

Developers should make changes in their main application to start the CLI listener. This CLI listener will listen for CLIs from the host CLI server, and dispatch them to the appropiate CLI actions. The following example illustrate how to start the CLI listener in the main application

package com.myApp1;

import com.cisco.aesop.apphosting.cli_distribution.CLIDistributionVM;

public class MyApp1Main {

   public static void main(String[] args) {
      System.out.println("MyApp1Main.main: called");
      CLIDistributionVM.startCLIDistributionVMThread("myapp1");
      
      int num = 0;
      System.out.println("MyApp1Main.main: entering loop");
      while(true){
         System.out.println("MyApp1Main.main: " + num++);
         try {
            Thread.sleep(10 * 1000);
         } catch (InterruptedException e) {
            System.out.println("MyApp1Main.main: done");
            System.exit(1);
         }
      }   
   }
}

A example Makefile to compile the main application. Notice that the main application now needs "cli_distribution_vm.jar" to compile. This jar file is found in the SDK, under sdk/jar/cli_distribution_vm.jar

JAVAC = javac

default: build_app

build_app:
mkdir classes/
$(JAVAC) -classpath cli_distribution_vm.jar com/myApp1/MyApp1Main.java
-d classes
mkdir jar/
jar cf jar/myApp1.jar -C classes/ com/

clean:
@rm -rf classes/ jar/ *~

Make sure that the third party application is included in the project source directory

-bash-3.00$ cd /tmp/proj_src/
-bash-3.00$ ls
app_bin cli_comm etc usr
-bash-3.00$ ls app_bin/
myApp1.jar

Packaging and bundling application with dependency on plugin CLI add-on

Finally, developers should use the packaging and bundling tool to package and bundle the application with dependency on the plugin CLI add-on package. For more information, please refer to the SDK packaging and bundling section.

Developers first make sure that under the project directory, a pkg/ directory is created, and it contains the plugin CLI add-on package

bash-3.00$ cd /tmp/proj/pkg/
bash-3.00$ ls
cli_plugin.0.0.2.45.prt1 cli_plugin.pkg

Then, developers can run the packaging tool. Notice it contains a dependency on the plugin CLI SSID

/tmp/sdk/tools/pkg_build.sh --project-dir '/tmp/proj' --dev-cert
'/tmp/devcert/dev_certificate.jvln'
--dev-auth '/tmp/devcert/dev_authorization.jvln' --private-key
'/tmp/devcert/private.key'
--name 'myapp1' --version '45.5' --source-dir '/tmp/proj_src' --deps
'b4b0ee92-cf8e-472b-8434-e8e7412ec71a,all'
--disk-limit 'min=5000M,opt=6000M' --memory-limit '128M' --cpu-limit
'min=4000,opt=5000'

Then, developers can run the bundling tool to bundle the third party application and the plugin CLI add-on together

/tmp/sdk/tools/pkg_bundle.sh --project-dir '/tmp/proj'
--private-key '/tmp/devcert/private.key'
--output 'myapp1_cli.pkg' '/tmp/proj/pkg/cli_plugin.pkg'
'/tmp/proj/pkg/myapp1.45.5.pkg'

The project pkg directory now contains the bundled application

-bash-3.00$ cd ../proj/pkg/
-bash-3.00$ ls
cli_plugin.0.0.2.45.prt1 myapp1.45.5.pkg myapp1_cli.pkg
cli_plugin.pkg myapp1.45.5.prt1 myapp1_cli.prt1

Install the bundled application

Developers can try install this package via the CLI:

se-1-100-50-147.unspecified> software install add url ftp://myworstation
/pub/apps/myapp1_cli.pkg ...

Run the main application, issue the CLIs

During installation, the blade should reboot. After rebooted, make sure that the third party main application, which runs the CLI listener, is started. The following startup command (run in the third party application vserver context) in Java serves as example of what library files and path are needed with starting the CLI listener in the main application

-bash-2.05b# cd app/
-bash-2.05b# ls
myApp1.jar
-bash-2.05b# java -cp ./myApp1.jar:/usr/lib/java/localsocket.jar:/usr/lib
/java/cli_distribution_vm.jar:/cli_comm com.myApp1.MyApp1Main

On the CLI console, the new CLIs should be available

se-1-100-50-147.unspecified> app-service custom myapp1
se-1-100-50-147.unspecified(exec-appservice-myapp1)> ?

end Leave app-service exec mode
math Math operations with Java actions
no Negate the command
se-1-100-50-147.unspecified(exec-appservice-myapp1)>
se-1-100-50-147.unspecified(exec-appservice-myapp1)> end
se-1-100-50-147.unspecified> config terminal
Enter configuration commands, one per line. End with CNTL/Z.
se-1-100-50-147.unspecified(config)> app-service custom myapp1
se-1-100-50-147.unspecified(config-appservice-myapp1)> ?

end Leave app-service config mode
no Negate the command
store Store a value
se-1-100-50-147.unspecified(config-appservice-myapp1)>

When user invoke the CLI, they should see the response from the CLI action:

se-1-100-50-147.unspecified> app-service custom myapp1
se-1-100-50-147.unspecified(exec-appservice-myapp1)> math 12 add 34
exec_math.eval: answer is 46.
se-1-100-50-147.unspecified(exec-appservice-myapp1)>

Config CLIs

Configuration CLIs are CLIs that can be persistently stored in the blade's flash memory. In the above example, "store key_1 <num>" is a config CLI. Executing config CLIs is similar to that of exec CLIs

se-1-100-50-147.unspecified> config terminal
Enter configuration commands, one per line. End with CNTL/Z.
se-1-100-50-147.unspecified(config)> app-service custom myapp1
se-1-100-50-147.unspecified(config-appservice-myapp1)> store key_1 34
se-1-100-50-147.unspecified(config-appservice-myapp1)>

Issuing a config CLI will not display the response, but it nevertheless will invoke the CLI action. Also, this CLI can be seen by doing "show run" in the app-service. User can do "write memory" to write this CLI to flash.

se-1-100-50-147.unspecified> app-service system myapp1
se-1-100-50-147.unspecified(exec-appservice)> show run
Generating running configuration:

app-service system myapp1
hostname se-1-100-50-147.unspecified
shutdown
status-monitor interval_threshold 12 recovery_threshold 5
exit
app-service custom myapp1
store key_1 34
end
se-1-100-50-147.unspecified(exec-appservice)>

The way to retrieve the config CLI the next time the blade is rebooted, or the application is resetted, is by using the application status notifier. Application status notifier is a command to signal the application health state (more info can be found in the "Health Monitor" section). Anytime the application status monitor is invoked to set the application to "ALIVE" state will triggered all the plugin config CLIs to be sent over to the CLI actions.

-bash-2.05b# /bin/app_status_notifier myapp1 ALIVE

APPLICATIONEXTENSIONPLATFORM:Return To Top

e. Health monitor

Checking and Changing Application Health

There is a CLI to display the health of application:

blade# appservice system
(myapp)# show state
APPLICATION STATE HEALTH
myapp online —

The column marks "HEALTH" indicates the current application health state.

There are 4 HEALTH states:

1. "---" default health state when application is reset or shutdown
2. "INITIALIZING" indicates application is starting up
3. "ALIVE" indicates application is up and running
4. "DOWN" indicates application is down

This application health state should be changed by application itself. For example, application can change it from "INITIALIZING" to "ALIVE" when it knows that all the initializing is done.

Application can use the command /bin/app_status_notifier to change the health state:

vserver-shell# /bin/app_status_notifier myapp ALIVE

Application Status Monitoring

Watchdog scripts

If the application state is "online" and application health is not "DOWN", the application will be monitored by the application status monitor.

• Application needs to provide one or more health check script(s)/executable(s) and bundled in their package, if it wants to leverage this application status monitor functionality. The number of scripts/executables is application dependent. Shell scripts and c executables are supported.
• Each application has its own way to determine whether the application is running or not. The logic of health check scripts/executables is application dependent. For example, it can be based on Process Identifier (PID), or some response to an application "ping".
• The watchdog script(s) need to be placed in the predefined directory, /opt/app_status_monitor/watchdogs/ in the application vserver directory, and needs to be executable. All watchdog scripts inside that directory will be invoked, and if application developer wants to ensure correct ordering for script invocation, they can follow the template watchdog script format, which is W**.sh, where ** is a two digit number. The template script is named W00template.sh and is provided for every vserver.
• Application status monitor has a heartbeat of 5 seconds, this is the minimum interval application status monitor performs any monitoring. Given a monitor interval of 12, the monitor will perform monitoring on each vserver every 12 heartbeat intervals (i.e. 1 minute). There is a CLI to configure this monitor interval. For example, specifying a value of "3" for this vserver means that application status monitor will wait for every 3 heartbeat to monitoring this vserver.
• The script(s) or executable(s) need to return the status code. A zero (0) status code indicates the application is healthy and alive. A non-zero status code indicates to the monitoring system, the application is not functional (i.e. crashed etc.). Example of a watchdog script:

#!/bin/bash
if [APPLICATIONEXTENSIONPLATFORM: -e /var/run/myapp.pid ]; then
exit 0
else
exit 1
fi

• When any one watchdog returns a non-zero status code, information of this failed watchdog will be logged, such as name of watchdog, return status, and the time of failure. There is a recovery counter that counts how many times failure occurs, and it works like a delay in taking any actions. A recovery counter of 3 means that the application monitor has run the monitoring for 3 iterations and was either getting non zero return status, or the watchdog has been running for over 3 monitor interval and is not returning. There is a configurable recovery threshold that decides how many recovery counter before taking the next action. When the recovery threshold is reached vserver will be restarted.
• After vserver restarted, application status monitor shall continue to run, and the steps described above will be repeated. There is no limit on how many times the vserver restarts.

CLIs and configurations

There are CLIs to show the status monitor, as well as configure the monitor interval and recovery threshold.

1. Show status monitor

blade# appservice system myapp
(myapp)# show status
Application: myapp1
Monitor status: PASSED
Monitor in progress: No
Last executed watchdog: W01myapp1_test.sh
Last executed date: Tue Jul 10 10:22:06 PDT 2007
Last failed watchdog: W01myapp1_test.sh
Last failed return code: 4
Last failed date: Mon Jul 9 12:34:18 PDT 2007
Last restarted date: Mon Jul 9 12:33:32 PDT 2007
Recovery threshold: 6
Monitor interval: 2

2. Change the monitor interval and recovery threshold

blade# config terminal
(config)# appservice system myapp
(config-myapp)# status-monitor monitor_interval 24
recovery_threshold 10
(config-myapp)# end

In addition, third party developers can provide a default configurations via a config file packaged together with the third party application. This file name and path is /opt/app_status_monitor/config/config, and it should look like this:

monitor_interval 50
recovery_threshold 10

The two configuration values "monitor_interval" and "recovery_threshold", followed by a space, and then a number ranges from 1 to 99. Both must be present.

The CLI configuration values takes precedence over the config file values, since the config file only serves as a default at application is installed.

APPLICATIONEXTENSIONPLATFORM:Return To Top

f. SSH tunneling

Given that AXP supports vserver SSH connection, AXP also supports third party developers to utilize SSH tunnelling to enhance their applications. Third party developers can run applications such as X services that are tunnelled through SSH. In addition, developers and application administrators can use the SSH tunnelling to debug their own applications and vserver environment. Given that user can login to the vserver SSH server, we support two SSH tunnelling options

1. "ssh -L" - Port Forwarding
2. "ssh -X" - X11 Forwarding

Detail explanation on these two options are best found on the manual pages for SSH. The link below provides one example of online resources:

http://www.die.net/doc/linux/man/man1/ssh.1.html

Port Forwarding

One example to connect with "-L" port forwarding option:

workstation-shell# ssh -L :: @ -p
workstation-shell# ssh -L 7777:localhost:8888 tunnel_root@myblade -p 2022

X11 Forwarding

One example to connect with "-X" X11 forwarding option:

workstation-shell# ssh -X @ -p
workstation-shell# ssh -X tunnel_root@myblade -p 2022
vserver-shell# xterm &

APPLICATIONEXTENSIONPLATFORM:Return To Top

g. Utilizing the Promiscuous packet API

NAM - Network Analysis Module

Overview

Currently Cisco IOS enables a configuration to duplicate packets for network analysis when it detects the presence of the network analysis module (NAM) or intrusion detection module (IDS). Cisco IOS is notified through a card type cookie which consists of the product ID and a controller ID. The Cisco ISR service module cookie is similarly programmed to contain the same controller ID as a NAM to enable the Cisco IOS packet analysis CLI to be turned on for the service module.

How to configure NAM

Under the interface you would like to monitor traffic, configure analysis-module monitoring. Both incoming and outgoing traffic are monitored.

For example:

interface GigabitEthernet0/0
description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$
ip address 1.100.50.221 255.255.255.0
duplex auto
speed auto
analysis-module monitoring
no keepalive
end

Advantages of NAM

• Easy to configure
• Lightweight Export.
• Exports BOTH inbound/outbound traffic on the configured interface
• Exports locally generated packets (router generated) (i.e. ping reply from the ISR)

Limitations of NAM

• No ACL support
• No Sampling rate support
• Can only export to an attached NAM or IDS module. (sufficient for AXP).

RITE - Router IP Traffic Export

Overview

RITE is a lightweight mechanism for duplicating and exporting monitored traffic on an interface to an analysis module. The analysis module need not be attached to the ISR (unlike NAM), they can be connected over a physical interface with different encapsulations such as Ethernet, PPP, and many more. The Integrated-Service-Engine interface used by the AXP service module behaves like an Ethernet interface and is therefore a supported export interface for RITE.

RITE provides a more granular traffic capture over NAM. One can specify ACL, sampling rate, and also the direction of the capture to export more specific traffic on an interface. There is however one limitation on RITE; router generated traffic will not be captured by RITE. Router generated packets are process switched and not cef switched, and is not along the capture path of RITE.

How to configure RITE

The high level steps to configure RITE are:

1. Create a RITE capture profile
2. Configure the parameters of the profile
3. Apply the profile on the interface one wishes to monitor

Create a RITE capture profile

Create a profile with ip traffic-export profile [APPLICATIONEXTENSIONPLATFORM:any name]

Configure the interface you wish to export to. This should be interface Integrated-Service-Engine 1/0.

Configure the mac-address of the Integrated-Service-Engine1/0 interface with the mac-address [APPLICATIONEXTENSIONPLATFORM:address] command.

One can optionally configure bidirectional if one wishes to capture both inbound and outbound traffic. Inbound only capture is default.

A Note on the mac-address: Conceptually, the mac-address should be the valid mac-address of the service module. However, since the service-module is configured with promiscuous mode, invalid mac-address will also work. There is however still a problem since the service module is set to route packets by default, configuring a valid mac-address will actually cause the routing stack of the service-module to route the traffic which is not a desired behavior (this will cause duplicated packets to be routed). So to eliminate this problem, it is recommended that you configure an invalid mac-address (something other than the mac-address of the service-module - such as a.a.a), this way your traffic will reach the service module but will not be routed.

Example:

jc1(config)#ip traffic-export profile rite-bi
jc1(conf-rite)#?
IP traffic export profile configuration commands
bidirectional Enable bidirectional traffic export
exit Exit from ip traffic export profile sub mode
incoming Configure incoming IP traffic export
interface Specify outgoing interface for exporting traffic
mac-address Specify ethernet address of destination host
no Negate or set default values of a command
outgoing Configure outgoing IP traffic export

ip traffic-export profile rite-bi
interface Integrated-Service-Engine1/0
bidirectional
mac-address a.a.a
!

Configure RITE export parameters

One can configure incoming or outgoing parameters by entering each menu. You can apply a regular access-list, or configure the sample rate of the export traffic.

Example:

jc1(config)#ip traffic-export profile rite-bi
jc1(conf-rite)#incoming ?
access-list Apply standard or extended access lists to exported traffic
sample Enable sampling of exported traffic

ip traffic-export profile rite-bi
interface Integrated-Service-Engine1/0
bidirectional
incoming access-list 4
outgoing access-list 4
mac-address a.a.a
incoming sample one-in-every 2
outgoing sample one-in-every 2
!

Apply the profile to an interface

Configure ip traffic-export apply [APPLICATIONEXTENSIONPLATFORM:profile name] on the interface you wish to monitor.

Example:

interface GigabitEthernet0/0
description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-GE 0/0$
ip address 1.100.50.221 255.255.255.0
ip traffic-export apply rite-bi
duplex auto
speed auto
no keepalive
end

Advantages of RITE

• Lightweight export
• Can capture incoming or bidirectional traffic
• Allow user to configure ACL
• Allow user to configure sampling rate
• Can export to interfaces with different encapsulation types (though for AXP, we're only interested in the Integrated-Service-Engine type).

Limitations of RITE

¿ Due to design of RITE, router generated packets cannot be exported since it does not go through the CEF switching path. (i.e. If someone pings the ISR, the ping reply that the ISR generates will not be exported even if you have bidirectional configured). If capturing this traffic is important for your application, consider using NAM.

How to decide whether to use NAM or RITE for your application

Both packet monitoring features have its pros and cons. The advantages and limitations for each is listed in their respective sections. Here is an attempt to provide some suggestions on which to use for your application:

• If your application requires monitoring router-generated packets then use NAM as RITE cannot support this.
• If you just want to quickly configure something and get it to work, use NAM.
• If you want more granular control on your exported traffic, then RITE would be the choice for you. Keep in mind the above limitation. In general, if you don't care about router-generated packets I would recommend RITE as it provides more feature set.

Note: Both features are allowed to be configured at the same time, however, it is not recommended to do so. Each feature does its own duplication, you will receive two duplicated packets if you configured both of them at the same time.

Packet Capture API support:

• Refer to section on NAM and RITE for configuration of packet diversion technology that will capture and divert replicated packets to the AXP blade.

• In terms of API support, AXP enables application to capture packet through the use of promiscuous mode thorough the use of raw socket the following kernel configuration have been enabled for packet capturing:
  o raw socket interface (CONFIG_RAW) - http://en.wikipedia.org/wiki/Raw_socket
  o raw socket memory mapped mode (CONFIG_RAW_MMAP) -
  o socket filtering (CONFIG_FILTER).

These features can be accessible directly by 3rd party developer.

• AXP also provides the libpcap library as higher level abstraction for packet capture.

• libpcap references:
  o Libpcap library(http://www.tcpdump.org)
  o "man 3 pcap" for further details.
  o http://www.cet.nau.edu/~mc8/Socket/Tutorials/section1.html

• A cavaet to this is that the linux-vserver technology does not fully virtualize networking interfaces as a result, with raw socket being used, all available network interface created under AXP can be monitored by the an application running inside a vsever container.

Example of using Raw Socket

s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
struct sockaddr_ll socket_address;
socket_address.sll_family = AF_PACKET;
socket_address.sll_protocol = htons(ETH_P_ALL);
bind(s, (struct sockaddr *) & socket_address, sizeof(socket_address));
recvfrom(s, buffer, ETH_FRAME_LEN, 0, NULL, NULL);

APPLICATIONEXTENSIONPLATFORM:Return To Top

h. Logging support and recommendations

Virtual Instance Logging Support

Syslog is used as the tracing/logging facility for the applications inside the virtual instance (VI).

1. Configure Application Logging Facility

• syslog-ng, an improved syslogd daemon, is used for VI logging service.

• The syslog file, /var/log/messages.log, is pre-configured by the system.

• AXP syslog supports file rotation - rotation of two files is supported.
  o messages.log (primary log file)
  o messages.log.prev (backup log flie)
  o User is able to configure the log file size limit. Once /var/log/messages.log reaches the configured limit, its contents is moved to a backup log file, messages.log.prev. A new /var/log/messages.log file is started.
  + conf t
  + app-service [APPLICATIONEXTENSIONPLATFORM:appname]
  + limit log-file size [APPLICATIONEXTENSIONPLATFORM:MegaBytes] ( default 5MB)

• AXP syslog supports syslog levels
  o Even though all log levels are support, but CLI reduces configuration to 4 log levels.
  + conf t
  + app-service [APPLICATIONEXTENSIONPLATFORM:appname]
  + log level [error ] (default warning)
  + error - Events with LOG_ERR, LOG_EMERG, LOG_ALERT, and LOG_CRIT are logged.
  + warning - Events with LOG_WARNING, LOG_ERR, LOG_EMERG, LOG_ALERT, LOG_CRIT are logged.
  + notice - Events with LOG_NOTICE, LOG_WARNING, LOG_ERR, LOG_EMERG, LOG_ALERT, LOG_CRIT are logged.
  + info - Events with LOG_INFO, LOG_NOTICE, LOG_WARNING, LOG_ERR, LOG_EMERG, LOG_ALERT, LOG_CRIT are logged

• AXP syslog supports remote logging
  o Logging support is default to log locally to /var/log/messages.log file.
  o User is able to configure the logging to remote syslog server.
  + conf t
  + app-service [APPLICATIONEXTENSIONPLATFORM:appname]
  + log server address [APPLICATIONEXTENSIONPLATFORM:a.b.c.d] (default disabled)

2. Application Logging Usage

• Third party applications can use either the C syslog API or Java log4j Syslog Appender for logging. As mentioned, log messages goes to /var/log/messages.log

• Syslog for C
  o See syslog API

*
o Examples:
+ setlogmask (LOG_UPTO (LOG_DEBUG));
+ openlog("myapp", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_LOCAL1);
+ syslog(LOG_DEBUG, "myapp_user_management.eval() processing CLI message\n");

• Log4j for Java
  o See Syslog Appender in the Log4J document
  o If an application uses log4j, it might use multiple appenders for the logging. To leverage AXP logging facility (syslog), application need to specify the Syslog Appender to log to syslog.
  o SyslogAppender can specify its own log level in the log4j.properties file. The log will first be filtered by the SyslogAppender level configuration. Then, the log will be send to the syslog on the vserver, which will then be filtered by the syslog level configuration (i.e. log level [APPLICATIONEXTENSIONPLATFORM:level] CLI as above). Hence, both level configuration is in effect, and the logs are filtered first by the log4j then by the syslog.
  o Currently, if an application chooses to use the AXP syslog mechanism, they can set the log level in its log4j SyslogAppender to be of lowest priority (e.g. DEBUG), then control the filtering through the syslog CLI on AXP.
  o In the future release, AXP will provide plugin for application to dynamic change Log4J syslog appender to the log level configured via CLI.
  o Examples: log4j.properties to enable Syslog Appender:
  + # Set root logger level to DEBUG and its only appender to A1.
  + log4j.rootLogger=DEBUG, mySyslog
  + # mySyslog config
  + log4j.appender.mySyslog.layout=org.apache.log4j.PatternLayout
  + log4j.appender.mySyslog.layout.ConversionPattern=%p %t %c - %m%n
  + log4j.appender.mySyslog.SyslogHost=localhost
  + log4j.appender.mySyslog.Facility=USER
  + log4j.appender.mySyslog.threshold=INFO
  o (new) For any APIs and libraries provided by AXP, logs are redirected to stdout (via System.out.println). If an application that uses these APIs and libraries and wants to redirect such logs to syslog, the application can be executed with its output piped to /bin/logger. Given the example of starting an application that uses CLI Plugin

1. java -cp ./app_bin/myApp.jar:/cli_comm/:/usr/lib/java
   /localsocket.jar:/usr/lib/java/cli_distribution_vm.jar
   com.myApp.MyAppMain | /bin/logger -p info

This will redirect stdout of the CLI Plugin (as well as any System.out.println called by the application itself) to the syslog as priority INFO

3. Application Logging Viewing

• CLIs are provided to display the contents of all log files under /var/log directory in the Vserver context.
  o >app-service [APPLICATIONEXTENSIONPLATFORM:appname]
  o exec-appname> show logs
  + Show all the files under /var/log directory
  o exec-appname> show log name [APPLICATIONEXTENSIONPLATFORM:log name]
  + show the content of the spcified log file
  + wild card is supported for the file name (i.e. *.log)
  + output modifiers (i.e. paged, containing, tail, |, and less) are supported for easier viewing

APPLICATIONEXTENSIONPLATFORM:Return To Top

i. Creating a application console within the CLI

Overview

A CLI command, connect console , is provided to allow application to create console access in its own vserver instance (Merwan: please use consistent term to name a Vserver through out the document. Whatever term is decided). Application can leverage this capability to

• Access the shell inside its own vserver context, or
• Launch their own CLI console access, or
• Run an application program inside vserver

Usage

1. To leverage this capability, application needs to provide a "console" file which can be a executable or a script. Please see the sample files below.
2. The file needs to be placed under /bin directory in the source files to be packaged. (i.e. /bin/console)
3. After application is installed, execute the connect console command under the application context. This will trigger the execution of /bin/console file.

• >app-service sampleApp
• exec-sampleApp>connect console

Sample

The following provides some sample /bin/console files for references:

• Used to access the shell
  o One way is to link the bash program to this file using post-install script or application startup script
  + ln -s /bin/bash /bin/console
  o The other way is to have one line in the /bin/console file as follows:
  + #!/bin/bash
  + bash
• Used to cross connect to application CLI console
  o #!/bin/bash
  o telnet 127.0.0.1 34567
• Used to run a application program
  o build application program as executable and named as console, or
  o launch the application program in the console file
  o #!/bin/bash
  o start_myapp.sh

Linux Shell access via CLI

Another optional packaged CLI command, linux shell , is also provided to allow application to access the guest os shell.

• This CLI is bundled in infrastructure add-on package, app_debug.pkg.
• After intalling the package, user can go into linux shell by doing the following:
  o >app-servcie [APPLICATIONEXTENSIONPLATFORM:appname]
  + linux shell

3. Application debugging

a. Vserver commands

• Not Sure what is required here we do not expose Vserver command in GuestOS?. We have provided wrappers in CLI

APPLICATIONEXTENSIONPLATFORM:Return To Top

b. GDB Debugging

Set up GDB Debugging Session

GDB debugging capability is provided through the "gdbserver" and "gdb" commands. The GDB connection is done through the vserver SSH connection. The steps are:

1. Setup a debugging session on the application running in the vserver, using "gdbserver"
2. Make a SSH tunnel connection ("-L" option for port forwarding)
3. Run "gdb" on the workstation and connect to the SSH tunnel

Set up debugging session

There are two ways to setup gdb debugging session

1. Login to vserver SSH server as "tunnel_root" and run "gdbserver" manually. Given that the application name is /my_app/debug

workstation-shell# ssh tunnel_root@myblade -p 2022
tunnel_root@myblade's password:
vserver-shell# gdbserver localhost:2222 /my_app/debug
Process /my_app/debug created; pid = 29165
Listening on port 2222

2. Setup the "tunnel_user" startup script to do runtime "gdbserver" binding, using the application process ID

This requires changing the "tunnel_user" startup script, as example like this (given my application process name is "debug"):

#!/bin/bash
pid=`ps -ef | grep debug | awk '{print $2}'`
echo "gdbserver will be running on process ${pid}, binded to port 2222"
gdbserver localhost:2222 --attach $

Making SSH tunnel connection

Given that port 2222 is binded to the gdbserver on the vserver side, the following command will establish SSH tunnel from the workstation's port 4567 to blade's 2222

workstation-shell# ssh -L 4567:localhost:2222 tunnel_root@myblade -p 2022

Run "gdb" on the workstation and connect to SSH tunnel

Given that port 4567 is binded to the SSH tunnel on the workstation

workstation-shell-2# gdb
(gdb) file debug
(gdb) target remote localhost:4567
(gdb)

After that you can use gdb commands like "break", "continue", "step". Below list one online resource regarding GDB usage: http://www.cucy.net/lacp/archives/000024.html