Troubleshooting Guide

Application installation/deployment failed.

One of the major reasons for the application installation failure is not enough resources available on the device. Check the available space on the device using one of the clients below.

Using Local Manager

1. Check the available space on the device by clicking on "System Troubleshoot".

2. Look under Diagnostic summary disk usage. "/software" will tell you the available space on device for app installation.

Using ioxclient

Command: ioxclient platform diag

1. Choose summary.

2. Choose "n" on detailed output.

3. Look for "/software".

dev@dev-VirtualBox:~/Documents/iox/fd$ ixct plt diag
Currently active profile : 829
Command Name: plt-diagnostics
summary
memory
disk
process
networking
application
all Choose the diagnostic [1-7] : 1
Do you want detailed output (y/n)?n 
uptime: invalid option -- 's' Try 'uptime --help' for more information. 
eid: iox-ir809-11 pfm: IR809G-LTE-GA-K9 s/n: JMX2020X021 
boot: time: 2017-11-14 21:54:06 load: 21:54pm up 5 days 19:07, 1 user, 
load average: 0.00, 0.00, 0.00 
images: DevOS: unknown, Lnx: 1.4.16.4 .4, 
IOx: 1.5.0:develop:147bfca memory: ok, used: 96/936 (10%) 
disk: ok, used: /:4532/479740 (0%), /software:478776/765260 (62%) 
process: ok, running: 6/6 networking: ok 
logs: warning, errors: caf (0), tpmc (0), mhdserver (12) 
apps: warning, ac (D) dd (S?) lxcapp (S) nt02Test (R) 
simulatedVisulaization (S?) testApp (D) testLxc (D)

Using Fog Director

1. Go to devices page, and click on the device from the list. The available space is shown in a bar chart on that page (more information is displayed when the mouse cursor hovers on the chart).

If the resources requested are larger than what is available, the deployment fails. The application size includes the tar ball size. You must take into account the size of the untar components of the package.

2. If there is space available on the device, wait for the given amount of time for the installation to complete. Time varies on the size of the application.

The following table shows the maximum size of the application allowed on a specific device and the ideal time taken for app deployment on the following platforms based on the app size.

In good network conditions:

For IR800 platforms, use the following the command:

IR800#guest-os 1 disk-repartition ?
  1  disk1: 500MB vs disk2: 1800MB (NOTE : Actual storage available for apps will be less than the value chosen for disk2 for all profiles.
     See documentation for more info.)
  2  disk1: 700MB vs disk2: 1600MB 
  3  disk1: 900MB vs disk2: 1400MB 
  4  disk1: 1100MB vs disk2: 1200MB 
  5  disk1: 1300MB vs disk2: 1000MB 
  6  disk1: 1500MB vs disk2: 800MB 
  7  disk1: 1700MB vs disk2: 600MB 
 
IR800#

NOTE: This operation completely reformats the IOx VM, so you must reload the IOx (guest-os) VM. Then, you must reload and reactiviate all applications, including all application data stored in “/data”.

To reload, use the following command:

IR800#guest-os 1 image install flash:<Fog-Node-Software.bin> verify

Or, repeat the bundle installation using the following command:

IR800#bundle install flash:ir800-universalk9-bundle.SPA.<IOS-Version-Number>.M.bin

Application is running but not accessible.

If an application was activated in Bridge mode, check if the application has an IP address.

To check the IP address:

Using Local Manager

1. Click manage on the app.

2. Go to app-info, and click on interface-name eth0.

The network information on the pop up should display the IP address.

Using ioxclient

CMD: ioxclient app info

Look under "network info".

Using Fog Director

Go to devices page, and click on the device from the list. The application IP address is shown in the "App Details" section for that application on the page.

If the IP address is not assigned, you can use one of a few methods to debug the application based on the type of the application.

One method is try to assign a static IP address, and check if the application is able to get an IP address. Use one of the following methods:

Using Local Manager

1. Stop the application, and deactivate the application.

2. Click on "activate" on the activation page under Network Configuration.

3. Click on "Interface Setting".

4. In the pop up choose static. Fill in the required details and confirm.

5. Start the application.

Using ioxclient

1. Stop the application.

2. Deactivate the application.

3. Activate the application using a payload which has details of the static IP address.

Using Fog Director

1. Go to devices page.

2. Click on the device from the list.

3. Find the section for that application, and select the tab "EDIT APP SETTINGS".

4. Configure static IP address, and submit the new configurations.

LXC Applications

If the application does not have an IP address:

1. Check the app configuration, and verify if the app is in Bridge or NAT mode.

2. If it is in Bridge mode, check the DHCP server configuration. Log in to the application console, and check the ps output.

3. Check that DHCP client is running. If IOS is configured as the DHCP server, turn on debug in IOS to see the DHCP server logs.

Docker Applications

If the application does not have an IP address:

1. Check the IOS side GOS Pool Configuration. Use the show ip dhcp binding command from the privileged exec prompt (#).

2. Check the watchdog log of the application.

Networking Debugging

An application can be activated in two ways: NAT mode or Bridge mode (default).

In NAT mode IOx assigns an IP address to the application. In Bridge mode, you can assign the IP address in two ways: Dynamic or Static

In a Dynamic way, the application should get an IP address dynamically from IOS.

In a Static way, the developer has to assign an valid IP address.

Application goes to Stopped mode as soon as it started.

To debug this application:

1. Deactivate the application.

2. Activate the application in debug mode, and start using ioxclient.

Command to activate the application in debug mode is:

$: ioxclient app activate <app name> --debug on

This allows the application to stay in running state. Use the application console command to console in to the application.

Check if the required handlers are up.

How to access application console.

If application is of LXC or VM type, use the following console command to get into container namespace:

$: ioxclient app console <app name>

If application is of Docker type, use the following command to get into container namespace:

$: ioxclient app session <app name>

Log Files

Use the log files at any time to see if there are any errors with the application or system.

On Local Manager, click on the "System Troubleshoot". All the errors recorded in the CAF log will be available with details on the left pane. On the right pane you have the option to download the log for further analysis.

Cisco Application hosting Framework (CAF) is the key component of IOx that manages the operations of the applications on an IOx device.

CAF logs contain information relevant to the results of application state changes (start, stop, activate, deactivate).

Common platform logs syslog, dmesg are available to check the system status.

CAF Error Message Description