Troubleshooting Issues
| Device Status | Event Logs Message | Troubleshooting | Platform |
|---|---|---|---|
| Unheard | None | Check https://software.cisco.com/#pnp-devices, * Make sure device PID and S/N is matching * See if device has been redirected recently |
All |
| None | Connect to console port using mini-USB cable and check its status. * Device should be in PnP state (asking for Initial Configuration y/n). * If not, "enable" and "dir flash:", make sure the image you want to boot is listed first. "erase nvram:" and "reload" without saving config. |
IR800, IR1100 | |
| None | Check IP connectivity * If using Cellular as uplink, make sure it has the correct APN * ping 8.8.8.8 * telnet devicehelper.cisco.com 443, connection should "Open" |
All | |
| Bootstrapping | Device is Bootstrapping | Make sure the Configuration you used is correct. You may need to create a new Configuration Group to pick up latest bootstrapping configuration. | All |
| Device is Bootstrapping | Connect to the GW console. Make sure there is network connectivity (ping). * If not, check the Bootstrap Configuration in the group. It is possible that the Bootstrap Configuration caused the device to drop offline. |
All | |
| Device is Bootstrapping | Check if Tunnel1 is up and has an IP address. * If not, troubleshoot VPN issues. |
IR800, IR1100 | |
| Device is Bootstrapping | "show cgna profile all" * There should be only cg-nms-register, if you see cg-nms-periodic and device state is Bootstrapping, there must be some strange issue * Check that the url contains part of the cluster name. If not, this configuration is for a different cluster * Check Last Success / Last Failure. * If the last event is a failure, * see if it can resolve the hostname in cgna profile * "telnet <hostname> 443 /source loopback1", the connection should be Open. Otherwise, check why it is not able to connect. * debug cgna logging all, cgna exec profile cg-nms-register, look for return codes. * If there is no return code, likely there is a timeout. * If rc is 401, look for logs in the server. * If the last event is a success, there should be additional logs in the server related to error reaching the device. * If the error shows SSL, check that the device has "ip http secure-trustpoint CISCO_IDEVID_SUDI" in its config and it is an IR1101 running 17.2.1+ image. If yes, remove that line from the Bootstrap Configuration and re-register. |
IR800, IR1100 | |
| Error encountered during bootstrap process. | * Console into GW and "show run | s archive". If no archive config, add the following: archive path flash:/ max * Alternatively, simply delete and re-add the device and reset the device |
IR800, IR1100 | |
| Error pointing to Bootstrap Configuration | Check the corresponding line of Bootstrap Configuration | All | |
| Bootstrapping, After "Registration request from device." | SSL | From enabled mode, "debug ip http ssl error". See if there is SSL error within 3 min. If yes, troubleshoot certificate related issues. | IR800, IR1100 |
| "Connection reset" | Check to make sure the IP address is correct. The older cluster requires users to enter the GW IP manually. This could lead to multiple GWs using the same IP address blocks. Pick a different IP. | IR800, IR1100 | |
| "Error occurred while performing file upload operation" | "show ip http server status". See if self signed certificate is used. If yes, you may need to update your Bootstrap Configuration. IR800 needs to use the SUDI certificate. Verify by manually adding this config: ip http secure-trustpoint CISCO_IDEVID_SUDI You will need Bootstrap Configuration version v2.50+. If you are already using Bootstrap Configuration v2.50+, see if you have the following error in the device console: Starting File System integrity check Note: File System will be rebuilt If yes, this is a platform issue that needs to be addressed. Try "wr mem" and reload. |
IR800, IR1100 | |
| "invalid configuration commands" ... "snmp-server host <host> ... priv xxxxxx". | It is likely "<host>" is not resolving. Fix it by adding "ip host <host> <ip>" in Bootstrap Configuration. | IR800, IR1100 | |
| Up | AP Registration Failed | Check that the AP has the correct IP * ping vrf ciscoiot <ap-ip> (<ap-ip> is Loopback1 IP + 5) |
IR829 |
| AP Registration Failed | Session in to AP * service-module wlan-ap0 session * AP prompt should be "ap>" * enable password should be "Cisco" * "show ip int br" should 1 interface with AP IP * If AP looks like it was used by someone else, check flash: for any .cfg file and remove them |
IR829 | |
| AP Registration Failed | Check AP mode by logging into AP. AP can only be in Autonomous Mode. If AP is in Unified/Light-weight Mode, fix it first. | IR829 | |
| Down | AP Registration Failed | * Check if the device has an IP address using the IOS show command - show ip interface brief * Ping the device IP address to check reachability * Device may not have Down status on IoT OD. Use IOS Show commands to verify device status. |
All |
| Onboarding | Devices not successfully onboarded | * Is the device able to reach outside the network to the internet? Telnet to the US or EU clusters for port 443 and verify. If the connection opens, the device has access to the internet. * Check if the DNS server information is configured on the device. * DNS server should be able to resolve the public names respective to your cluster (US or EU). * Check if the required network ports and protocols are open on the firewall to allow access to IoT OD. (You can use: telnet us.ciscoiot.com 443 to verify if DNS can resolve the domain and connect over the internet.) * If the above requirements are met, make sure the device config is erased, as it will be provisioned through PNP. Reboot the device to initiate the connection again. |
All |
| Config Failure | * If device status shows Config Failure, click Edge Device Manager > Inventory > select a device > Event Log. * Find device in list to determine reason. |
All |
IMPORTANT:
- Only users with permissions of Manage Devices or Troubleshoot Devices can view the screen below and perform troubleshooting functions. For a complete list of Roles and permissions for each role, see Add and manage user access.
- Only devices in Inventory > In Use can be diagnosed with troubleshooting options.
Troubleshooting a device
- To troubleshoot a device, click Edge Device Manager > Inventory > select a device > Troubleshooting tab.
The following three Device options are available:
- Ping IP address from the device
- Traceroute IP address from the device
- IOS Show Commands
Each radio button option requires this information.
| Ping IP Address from device | Traceroute IP Address from the device | IOS Show Commands |
|---|---|---|
| IPv4 / Hostname * | IPv4 / Hostname * | Commands* |
| Set DF Bit in IP Header (Enable or disable) |
NA | NA |
| Datagram Size | TTL Maximum | No device config commands allowed |
| Source | TTL Minimum | No exec commands are allowed |
The * indicates required fields.
- Click Execute.
- Click Show Result.
Note: Request only one troubleshooting command at a time. Wait for the results before requesting another command.
IOS Show Commands
Whenever you run an IOS Show Command, it persists in the drop-down list as a "history." There is a maximum of 15 commands in the list in first out (FIFO) order. This Command history is shared between different devices of the same device type in a given organization. Pinning, unpinning, and deleting show commands from the history are available on the drop-down history, based on the user's role.
Tenant Admin role can perform the following:
- Pin a command so that it shows at the top of the history list. A maximum of 15 commands can be pinned.
- Unpin a command from the history drop-down list.
- Delete a command from the history drop-down list.
User with troubleshoot devices permissions can:
- Delete an unpinned command from the history drop-down list.
Note:
- All roles can run IOS commands.
- If a command is pinned, only Tenant Admins can delete that command from the drop-down list.
- You can only delete unpinned commands from the drop-down list.
- Externally-managed devices are supported.
- The drop-down list is device type + organization-specific information ONLY.
The screen below shows the Tenant Admin role privilege. The command history drop-down menu may show pinned (solid), unpinned (outlined), and delete icons.
IR829 Devices Only
For any IR829 device with Embedded Access Point, there is a new option to run troubleshooting show commands on the access point from IoT OD. To view the Access Point (AP) details, use the Show AP Command.
All of the above behaviors apply to the AP Show Commands for the IR829 devices. If you delete a show command from the drop-down list, you see a Warning dialog box. Click Confirm to delete a command.
Reboot the device
- Click Reboot and the device reloads.
- Click Show Result to see the status.
- To validate that device performed a reboot, click Event Log tab. Find messages: "Device Reboot Initiated" and "Device Reboot Success" with a time stamp.
Refresh Metrics
- Click Refresh Metrics. All the periodic metrics are refreshed using this command.
- Click Show Result to see status.
- To validate that device is showing very latest information, click Summary tab. In Device Details, find Last Heard information.
Note: To set periodic metric intervals, see Monitor Network Device Status.
Factory Reset
This commands resets the device to the factory settings.