Introduction

If the alert feature is enabled for the organization, the EI system generates events and alerts based on the status changes that occur in the system. Events are generated when the system detects an important status change. Some of these events can indicate an issue in the system. Alerts are generated when an issue continues. All alerts have configurable thresholds that can be used to fine-tune their behavior based on the system requirements. These detection thresholds help to prevent frequent closure and raising of alerts when the underlying status that causes or resolves the issue is temporary, such as when a patchy network connection causes connection loss and recovery between the EI agent and EI cloud.

Currently, the following alerts are supported:

  • EI Agent Offline
  • Data Delivery Impacted
  • Policy Unhealthy

Alert Details

Alert Name Description Recommendations
EI Agent Offline This alert is generated when the EI cloud loses contact with an EI agent for more than the configured threshold.
This alert is closed when the EI agent contacts the EI cloud and remains online for more than the configured thresholds.		 1. Check the connectivity between the EI agent and EI cloud by examining internet connectivity related events for this router in IoT OD and the routers log bundle.
2. Check if the device is running with a supported Firmware version.
3. Check the EI agent logs (broker.log) to see if there are any logs indicating connectivity issues if the device can be accessed using Local Manager through SEA.
Data Delivery Impacted This alert is generated when the EI agent fails to provision the new data fast enough and/or continuously queue up data due to exceeding data destination provisioning capacity.
This alert is automatically closed when the input data rate remains below the data destination provisioning capa city for a reasonable period.		 1. Compare the amount of memory and CPU shares allocated to this EI edge agent against the usage. Reallocate CPU shares as per the requirements.
2. To prevent excessive data ingestion, check the frequency at which you request data from your asset.
3. If you use custom-logic to process in-transit data, check if an in-memory buffer is growing too large, or if it is not being cleansed.
4. Check ingestion rates and throttling on your data destination that might prevent or delay data provisioning.
5. Compare the provisioned bandwidth against the volume of data to provision over time.
Policy Unhealthy This alert is generated when a deployed data policy is unhealthy in at least one of the EI agents.
The alert automatically closes when the deployed data policy is healthy across all EI agents.		 1. Check the details of the alert to see the most recent events related to the error. Check if there are any errors indicated in the event log.
2. View the policy status and find the EI agents for which the policy is indicating an error. To view the error details, check the policy status details for those EI agents.
3. Download the log bundle from the EI agent, extract it, and check the EI Application logs to see if there are any indications related to the issue.

Temporary issues may not trigger an alert, and refer events page for more details. The current status of EI agents and policies are displayed on the appropriate pages, and in case of issues, these pages will provide additional details and causes.

Check the following when high level issues are reported in the system:

  • EI agent and data policy runtime status page (drill down to the last level of details)
  • Depending on the error, check the configuration:
  • Data policy configuration
  • Asset configuration and mapping
  • Asset Type Data definition model
  • Destination configuration

Troubleshooting Tips

General Errors

Agent Onboarding Issues

  • EI agent is not reachable

    Occurs when: Either

    • a wrong target tenant URL is provided, or
    • a wrong or expired token is provided, or
    • if the internet connectivity from the router is impacted, or the internet facing traffic is terminated by a firewall/proxy.

    Resolution: Make sure the tenant URL and token for the EI agent configuration are valid and double-check the internet facing connection.

  • EI Agent is unable to connect after successful deployment of the application and configuration within the expected parameters (token and broker URL). EI Cloud UI continues to report no new installations of EI.

    Occurs when: It might be connected to the use of a private cellular network (LTE or 5G) in this particular scenario. Based on the device and software defaults, the MTU may not be correctly configured for the EI agent if you are using a private cellular connection.

    Resolution: Change the IOS DHCP-Pool configuration to include DHCP Opt 26 hex and set the MTU to 1300.

    Device: IR1101 (other models might be impacted)

    Reference link: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_dhcp/configuration/15-sy/dhcp-15-sy-book/config-dhcp-server.html

    conf t <br>
ip dhcp pool ioxpool <br>
option 26 hex 0564.0000 <br>

    or
    Reference link: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipapp/configuration/15-sy/iap-15-sy-book/iap-tcp.html#GUID-E0CD16F9-E817-4B50-849A-4760E1C0A9B1 

    conf t<br>
ip tcp path-mtu-discovery age-timer 5 <br>

Troubleshooting Data Policies

Common Deployment Issues

  • Error "Asset error: configuring device"

    Occurs when: The message Asset error: configuring device appears as the first response when the device model is tested for the first time.

    Resolution: Wait for a few seconds, and the message disappears once the device configures.

  • Right type of assets are not mapped to the EI agent

    Occurs when: The device type and corresponding device instance are created, but the asset is not mapped to the connected router.

    Resolution: Make sure that the correct router is mapped to the device instance in the inventory listing.

  • When the communication from the device to the EI agent is impacted, or no data is received, or the device appears disconnected/offline

    Occurs when: Either the physical setup, network configuration, device type, or device configuration are incorrect.

    Resolution: Depending on the setup and connection type, there can be several reasons and also if the communication to this device on this router previously worked.

    Check if:

    • Appropriate physical wiring is used depending on the connection type. (current, voltage, and pin assignment on the connectors).
    • Correct network configuration and network accessibility from EI agent to subtended device.
    • The device's current operational state, the data that is available and exposed from it.
    • Configuration of attributes, polling frequencies, and data types based on device type and connection type are correct.
    • There are any error messages, check the EI agent logfile for the connector corresponding to the connection type to the asset. Increase verbosity if there are no errors and the previous checks did not show any root cause.

  • Communication between the EI agent and the data destination is impacted, and no data is received

    Occurs when: EI's destination configuration, or the network communication to the destination are all incorrect.

    Resolution: Depends on the data destination.

    Check network configuration and accessibility from the EI agent to the expected destination, as follows:

    • Go to destination configuration in the EI agent and check the target URL/IP and port. For example, you must check configured URL/IP, port, and topic at EI agent for MQ Telemetry Transport (MQTT).
    • Check destination configuration for correct resource provisioning, availability, and exposed interfaces/services. For example, the URL/IP address, port, and topic configured at the MQ Telemetry Transport (MQTT) destination must match the EI agent configuration.
    • Check if the destination is configured to handle the incoming provisioned payload. Compare the payload delivered by EI agent to the destination to ensure compliance.
    • Check the authentication and authorization mechanisms for the accuracy and validity of the certificates, keys, usernames, and passwords provided to the EI agent and destination.
    • Make sure that the network connection to the data destination must not be terminated by a firewall or proxy. Check the connectivity after configuration to maintain the integrity of the payload and communication encryption and to ensure that the destination information given is reachable.

  • Fails to undeploy a data policy or data rule

    Occurs when: There is an issue with the communication to the EI agent.

    Resolution: Undeploy the data policy or data rule again and it will be removed in the subsequent attempt.

Troubleshooting Data Logic Scripts

Common Data Logic scripting issues

If Data Logic script issues occur, check the following:

  • Make sure the Asset Type name is the same as the name used in your script.
  • Map exactly one asset instance to the EI Agent for each Asset Type used in the Data Logic.
  • Make sure the EI Agent where the Data Logic is deployed has Agent Status "Online".
  • Check the format of the "Output Logic Data Model" - [{"key":"sum", "type":"INT", "category":"TELEMETRY"}].
  • Supported data types are – INT, STRING, BOOLEAN, DOUBLE, BINARY.
  • Supported categories are - TELEMETRY, PROPERTY, ATTRIBUTE.
  • The Cisco EI VS Plugin logs can be viewed in OUTPUT tab by selecting the Cisco EI in the VS IDE.

Errors when executing Data Logic script

  • Error: identifier ‘my_modbus_asset_type’ undefined

Occurs when: If the Data Logic script uses the wrong Asset Type name, an error occurs when the Data Logic is deployed.

For example, if the input Asset Type variable name is “my_modbus_asset_type” but "my_asset_type.speed" is used in the Data Logic Script.

Resolution: Make sure the Asset Type name is the same as the name used in your script.

  • Error: identifier ‘output’ undefined

Occurs when: This error occurs if the wrong output function is used when a RawMode custom Data Model is included in the JSON script.

Resolution: Use the function publish(“output”,outputJsonVariable); in your script. This error occurs if the function output.publish() function was used.

  • Error: identifier ‘publish’ undefined

    Occurs when: This error occurs if the wrong output function is used when an Output Logic Data Model is used.

    Resolution: Use the function output.publish() in your script. This error occurs if the function publish(“output”,outputJsonVariable); was used.

Errors when Debugging Data Logic

  • Error: No asset of type sliding_average is mapped to selected EI Agent

Occurs when: All selected Asset Types should be mapped to the selected EI Agent. If not mapped then this error occurs.

Note: In the error message sliding_average is the Data Logic name.

Resolution: Map exactly one asset instance to the EI Agent for each Asset Type used in the Data Logic.

  • Error: Error while invoking action '/downstream/ei-gateay-1/downstream/edge-management/data_pipelines/add_pipeline' with parameters [rtid, data]: error (link is disconnected)

Occurs when: The selected EI Agent Status is "Offline" and "Policy Status is Unknown".

Resolution: Make sure the EI Agent status is "Online".

Advanced Troubleshooting

If the issue cannot be resolved based on the information available in the EI Cloud, the EI Agent logs can be collected and analysed

The EI Agent logs can be fetched in following three ways

  • from EI Agent Troubleshooting page/tab - this downloads the complete technical support log bundle. Once the file is extracted, the different logs like CAF logs, Application logs etc can be checked to find the root cause of the issue
  • from the Application Management of the IoT OD. Click on Applications Tab. Go to the instance page of the application, find the entry for the device having issue, and click View Application Logs. From this page, you can check the last few lines of different logs and also download any specific log file that is required for troubleshooting (broker.log or cpp-< link_name >.log for link specific logs)
  • from Local Manager(LM)

Log files present in an EI Agent

Broker.log

  • To check if EI Agent is connected to upstream successfully with valid EI Agent token and URL configured. This info confirms it - "Connected to upstream, https://-:443/conn".
  • If packet drops are observed at the destination , verify that the edge broker, ingress and egress links do not flap and in connected state. This info confirms links are connected- "LCM, DSLink < link_name > changed status to connected". Look out for unexpected restart of links - "LCM, Restarting DSLink < link_name > after unexpected stop".

Cpp-opcua.log

  • Check the logs for OPCUA device connectivity issues.

Cpp-azure.log

  • Check the Azure link log when a device is not connected to the Azure. Message "INFO SDK(0): Success - Azure provisioning service status: PROV_DEVICE_REG_STATUS_CONNECTED for device: " confirms that the device has been connected successfully.

Cpp-normalization.log

  • Check the normalization logs for any policy deployment and undeployment issues

Cpp-mqtt.log

  • Check mqtt logs for mqtt ingress and egress connectivity issues.

Cpp-modbus.log

  • Check modbus logs if the modbus device is unable to connect to the edge broker

Cpp.resu41.log

  • Check logs for message sending information, broadcast immediately, and device status

Cpp-ntcip.log

  • Check logs for connectivity with device and correctness of creation process of SPaT and SRM/SSM messages

Cpp-eip.log

  • Check logs for troubleshooting if EIP devices is having connectivity issue and data loss

Cpp-serial.log

  • Check logs to determine if the serial port options are correctly synchronised to device. For example, port opened successfully with baudrate {}, databits {}, stopbits {} and parity {}

Troubleshooting using Verbose Logging

If the default logs from the EI Agent is not providing detailed information about the issues, then more detailed information can be generated using Verbose Logging:

  1. From the left pane, click EI Agents and click on the EI agent for which you want to troubleshoot.
  2. Click the Troubleshooting tab.
  3. Click Start Verbose Logging. A warning that initiating verbose logging could create large log files in the device and affect the overall EI system performance depending on the policy configuration.
  4. Click Continue. A success pop-up and a timer under Download EI Agent log appears.

    Note: The verbose logging is activated only for a maximum of 5 minutes.

  5. Repeat the operation for which you would like to receive highly verbose logging.
  6. Click on the timer to stop verbose logging.
  7. Click Download. The log file preparation will take up to a minute.
  8. Click on the prepared log file to download it and analyze to find the root cause of the issue.