DevNet Dashboards - Converged Availability Monitor

The "DevNet Dashboards - Converged Availability Monitor (DD-CAM)" project is an automation offer from Cisco Developer Relations. These projects are based on leading practices developed in engagement with customers and partners. This specific DD-CAM project is derived from work with our Network Operations Center (NOC) for our CiscoLive conference events in the US and Europe. We plan to release modules that can be used together to enhance network observability.

This specific project provides a multi-domain, converged availability dashboard that uses Element Management Systems (EMSs) and controllers as sources of truth for device inventory. You will not have to run a network scanning tool or utility to feed inventory for DD-CAM. The current modules that extract inventory for DD-CAM are:

1. Prime Infrastructure (provided for legacy support, as the EMS is being phased out)
2. DNA Center
3. ACI APIC controller
4. Wireless LAN Controller (WLC)

Each type of EMS/controller-specific source of truth has a Python script which reads its inventory and imports it into a MySQL database. A separate Python script extracts the converged device list from the database and feeds it to an optimized ping utility, fping, from the opensource project at https://fping.org/ A simple dashboard is then generated and published into an Apache webserver for view on a browser. This browser can be in a kiosk mode for a NOC or it can be easily viewed on a mobile device - phone or tablet.

The business driver for this was to bring multiple management tools' and controllers' inventory together into a converged availability dashboard without requiring yet another network discovery engine. Each of these management tools already act as a source of truth for their own domain, so we will use their sophisticated discovery and inventory management processes to fulfill our needs.

Technology Stack / Architecture overview

Below is the overall architecture for the project.

Other things you might include:

Status

v1.0 released to Automation Exchange
This is based on technology and processes used for over 7 years in the CiscoLive NOC.

Screenshot / Output

Use Case Description

This project solves Availability Monitoring issues when dealing with cross-domain management tools and controllers. While plenty of other open-source projects exist to show availability dashboards, few address the basic first step, getting the network devices into inventory. They leave that exercise to you. This project extracts the inventory from several supplied sources of truth, such as Prime Infrastructure, DNA Center and the ACI APIC controller. Multiple instances of each of these systems are allowed, for large environments.
One of the challenges this project addresses is how to ping monitor so many end-points. We're using the fping (fast ping) project to do optimized ping tests.
Since this project was intended to be modular in the importation of devices from the management tools and controllers, it can be extended to other systems like VMware vCenter (monitor VMs), NetApp ONTAP (monitor filers), Kubernetes, etc. The options are many. Check back as we add more sources in the future.

Installation

Two options are provided: self-directed installation with requirements OR Docker. Pick whichever suits your preference, security requirements and deployment capabilities.

When running as a self-directed installation with requirements, it is suggested to run this project in its own Linux Virtual Machine. Development was done on CentOS Stream 8.3, but releases at or above the 7 train or other Linux variants should be fine. CiscoLive NOC deployments have been implemented on Ubuntu 22.04 LTS.

Additionally, Apache and Python environments should be installed - your operating system package maintainer may have provided one in your installation. Our guidance is to keep the system-supplied version and install the latest Python and use virtual environments (venv) to maintain separation of environments from the core operating system. Our development was done on Python v3.9.1, but anything above v3.7 should be sufficient.

Local, Linux-specific Installations:

Step 1: Linux environment

Build a standard Linux VM with these initial characteristics [increase later if more performance is needed]
1 vCPU, 4 GB vRAM, 20 GB vHD
RedHat Enterprise Linux, CentOS and Ubuntu are fine. Development was done on CentOS Stream 8.

Step 2: Python

Ensure you have at least a Python 3.7 or higher environment. Virtual Environment (venv) support is suggested.

Step 3: Apache

Install Apache web server on Linux VM.
This Apache installation instruction is a good reference, if you need assistance.

Step 4: MySQL

Install MySQL database server on Linux VM. Development was done on MySQL Community Edition 8.0.21.
This MySQL installation guide is a good reference, if you need assistance.

Step 5: fping

Install the fping utility from this enhanced fping source.

$ git clone --branch json-output https://github.com/bonkf/fping.git

Step 6: DD-CAM software

Download the DD-CAM project

$ cd <your_local_project_directory>
$ git clone https://github.com/jasoncdavis/devnetdashboards-convergedavailabilitymonitor.git

Step 7: MySQL Database user and table work

Note the SQL for creating tables and users for the project in the mysql-table-ddl.sql file.
Ensure you have a sound password selection for the database user.

Docker Installations:

Ensure you have docker and docker-compose installed in your environment.

Download the DD-CAM project

$ cd <your_local_project_directory>
$ git clone https://github.com/jasoncdavis/devnetdashboards-convergedavailabilitymonitor.git

In the project's docker directory is the docker-compose.yaml file which defines the environment's service requirements. Essentially, Apache, Python and MySQL images are created. Remember your first time running 'docker-compose up' will require some time to download images from standard repositories.

For the docker image we will create two container volumes, web-data and mysql-data, which maintain the dynamic content for the Apache server and the MySQL database. They will persist data if 'docker-compose stop' or 'docker-compose down' is executed. Ensure you monitor the host system's file storage. Development was done with about 20G disk space and easily manages thousands of endpoint monitors.

Access the Python container from the host server running docker, with...

$ docker exec -it dd-cam_python_1 bash

The project files are in /project/code.

Configuration

An 'optionsconfig.yaml' file defines the Prime Infrastructure, DNA Center and ACI APIC controller IP addresses and authorized API user credentials. The supplied optionsconfig.yaml file is currently pointing to DevNet Always-On Sandbox systems. Please note, the devices extracted from those inventories are NOT generally reachable from the public Internet, so your dashboards will have a lot of RED until you reconfigure it to point to EMS/controllers that have access to your network devices.

If using the docker container version, copy the src/optionsconfig-docker.yaml to src/optionsconfig.yaml to use docker-specific directory defaults.

Edit the src/optionsconfig.yaml file to suit your environment's inventory sources. Multiple entries are allowed. Comment out entries that are not currently used, preserving them for future options. Enter each server's hostname/IP Address, authorized API username and password and any other parameters of interest.

Usage

The first step in using DD-CAM is to get device inventory extracted from your EMS and controllers defined in the src/optionsconfig.yaml file.
The script src/GetPrimeInfraDevices.py, performs device list extraction from Prime Infrastructure server(s) and imports them into the MySQL database, 'inventory' table.
The script src/GetDNACDevices.py, performs device list extraction from DNA Center server(s) and imports them into the MySQL database, 'inventory' table.
The script src/GetACIAPICDevices.py, performs device list extraction from ACI APIC controller(s) and imports them into the MySQL database, 'inventory' table.
The script src/GetWLCAPs.py, performs device list extraction from Wireless LAN Controller(s) and imports them into the MySQL database, 'inventory' table.

The next step in using DD-CAM is to ping and update the inventory and pingresults tables.
The script src/PingandUpdateInventory.py, extracts the device list from the MySQL database and submits the list of devices to fping. It also collects the results and updates the database.

The final step in using DD-CAM is to create the web pages (dashboards) from the ping results in the MySQL database.
The script src/CreateAvailabilityDashboard.py, creates the dashboard from the MySQL database results.

Run the following Python scripts at least once manually or schedule in a crontab, as needed, based on your rate of network device change. These scripts extract the network inventory and put them into the MySQL database.

$ python getPrimeInfraDevices.py
$ python getDNACDevices.py
$ python getACIAPICDevices.py
$ python getWLCAPs.py

Run the following Python scripts in a crontab, based on how often you'd like to update the dashboard - every 5 minutes, every minute, etc.

$ python PingandUpdateInventory.py && python CreateAvailabilityDashboard.py

A standard crontab model could be:

$ crontab -e
#minute hour day_of_month month day_of_week command_to_run
#Example of running every two minutes, every day
*/2 * * * * python PingandUpdateInventory.py  && python CreateAvailabilityDashboard.py

If you are extracting devices from your management tools/controllers that you can't or don't want to ping for availability, use the mysql shell to update the 'inventory' table. Specifically, set the do_ping column value to 0 (zero) and the endpoint will not be pinged.

host-vm$ docker exec -it dd-cam_db_1 bash
bash-4.4# mysql -u dddbu -p devnet_dashboards
Enter password: ~~ddcam4DevNet!~~
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 95
Server version: 8.0.33 MySQL Community Server - GPL
Copyright (c) 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> update inventory
  -> SET do_ping = 0
  -> WHERE mgmt_ip_address = '30.1.1.1';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql>

Alternatively, you could install MySQLWorkbench, as a graphical database administration tool, and perform the updates.

DevNet Sandbox

The following DevNet Sandboxes are referenced in the src/optionsconfig.yaml file for sample sources of inventory:

Prime Infrastructure v3.8 Always-on Sandbox [Sandbox environment was retired]

DNA Center v2.3.3.5 Always-On Sandbox

DNA Center v2.3.3.5 Always-On Sandbox - secondary

ACI Simulator v6 Always-On Sandbox

How to test the software

The scripts should generate entries in the mysql 'devnet_dashboards' database and 'inventory' & 'pingresults' tables. Additionally an availability.html file should be dropped into the Apache web server's publication directory, usually /var/www/html. Access with the IP address of your VM or docker container instance - eg.
https://192.168.1.100:8443/DevNetDashboards/DDCAM/availability.html

Known issues

None known at this time.

Getting help

For additional help, look to the DevNet Developer Support team. If major enhancements are requested, the Cisco Customer Experience (CX) team can be engaged for transactional assistance.

If you have questions, concerns, bug reports, etc., please create an issue against this repository.

Getting involved

How to contribute is found in the CONTRIBUTING file.

Credits and references

1. fping project
2. bonkf's implementation of the JSON-output idea I proposed.

Licensing info

See the LICENSE.

Use Case