Nick's OSPF TroubleShooter (nots)

A simple but powerful Ansible playbook to troubleshoot OSPF network problems
on a variety of platforms. It is simple because it does not require
extensive preparatory configuration for individual host state checking.
It is powerful because despite not having the aforementioned level of
granularity, it rapidly discovers the vast majority of OSPF problems.

Contact information:
Email: njrusmc@gmail.com
Twitter: @nickrusso42518

• Supported platforms
• Summarized-test cases
• Operations
• Variables
• Logging
• FAQ

Supported platforms

Today, Cisco IOS/IOS-XE, IOS-XR, and NX-OS are supported. Valid device_type
options used for inventory groups are enumerated below. Each platform
has a folder in the devices/ directory, such as devices/ios/. The
file named main.yml is the task list that is included from the main
playbook which begins the device-specific tasks.

• ios: Cisco classic IOS and Cisco IOS-XE devices.
• iosxr: Cisco IOS-XR devices.
• nxos: Cisco NX-OS devices.

Testing was conducted on the following platforms and versions:

• Cisco CSR1000v, version 16.07.01a, running in AWS
• Cisco CSR1000v, version 16.09.02, running in AWS
• Cisco CSR1000v, version 16.12.01a, running in AWS
• Cisco IOSv, version 15.6M, running on IOU
• Cisco XRv9000, version 6.3.1, running in AWS
• Cisco 3172T, version 6.0.2.U6.4a, hardware appliance

Control machine information:

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

$ uname -a
Linux ip-10-125-0-100.ec2.internal 3.10.0-693.el7.x86_64 #1 SMP
  Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
$ ansible --version
ansible 2.8.7
  config file = /home/ec2-user/racc/ansible.cfg
  configured module search path = ['/home/ec2-user/.ansible/plugins/modules',
    '/usr/share/ansible/plugins/modules']
  ansible python module location =
    /home/ec2-user/environments/racc287/lib/python3.7/site-packages/ansible
  executable location = /home/ec2-user/environments/racc287/bin/ansible
  python version = 3.7.3 (default, Aug 27 2019, 16:56:53)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

Summarized test cases

The following tests are run in sequence. Note that the exact items tested
varies between platforms since command outputs and feature sets also vary.
Administrative tasks, such as creating directories and logging on the control
machine, are not detailed here for brevity.

Per device testing

Ansible logs into each OSPF router for the purpose of collecting information
and validating its correctness based on a small amount of pre-identified
state configuration. As discussed in the "variables" section, some of these
tests can be skipped by modifying the appropriate key/value pairs.

The list of tests run on each specific device are enumerated under each
README.md file inside the subdirectories of devices/ correlating to
each unique device type.

Whole network testing

After individual routers are validated, additional tests based on the
aggregated data from all routers are run. It is possible to run these
tests on a per-host basis, but that would effectively cause the same test
to be run N times rather than one time.

• Ensure there are no duplicate OSPF router IDs. While it is technically
  possible to duplicate RIDs in different areas (sometimes), there is no
  legitimate reason to do it. This playbook always considers
  duplicate RIDs to be an error condition.

Operations

This solution uses a GNU Makefile to simplify setup and daily operations.
The following make targets are supported.

• make: Same as make test
• make test: Runs all testing (lint, unit, integ) in sequence
• make setup: Installs packages and builds the vault password file
• make lint: Runs YAML and Python linters along with Python formatter
• make unit: Runs function-level testing on Python filters
• make integ: Runs the test playbook (integration test)

Variables

The following subsections detail the different types of variables, their
scopes, and their purposes within the playbook.

Process-level

This playbook assumes that all OSPF routers are in a single process, and
if they are not, only a single process can be checked at a time.

Process-level variables differ between device types. For a list of supported
process variables, reference the individual README.md files in each
devices/ subdirectory correlated to each device type.

Area-level

This playbook allows an unlimited number of areas to be specified, each with
their own area-specific configuration. The playbook assumes that there are
no duplicate areas in the network. For example, while it is possible to have
two disparate area 1 sections of the network tied into area 0, this playbook
does not support it.

The top-level key is the area ID, specified as a string in the format
"area#" where # is the ID itself. For example: "area0" and "area51"

• type: The area type, specified as a string from the following options:
  "standard", "nssa", "stub". No other options are allowed, and
  area 0 must be type "standard". This key is mandantory.
• routers: The number of routers expected to exist in a given area,
  expressed as a positive integer. To disable this check, exclude this key.
• drs: The number of designated routers expected to exist in a given area,
  expressed as a positive integer. Note that DRs exist on broadcast-style
  network segments only, and are unnecessary on point-to-point links over
  broadcast media such as ethernet. To disable this check, exclude this key.
• has_frr: Boolean representing whether OSPF Fast Re-Route
  (also known as Loop Free Alternative/LFA) should be enabled
  (true) or disabled (false) for this process. This test checks for the
  basic enablement (or not) of this feature and not advanced derivates, such
  as remote LFA (rLFA) or topology-independent LFA (TI-LFA).
  To disable this check, exclude this key.
• max_lsa3: The maximum number of summary LSAs (LSA type-3) that should be
  present within an area. This inclusive upper bound enforces a limit on
  the number of LSA3 for the purpose of flood reduction and memory
  consumption. It can also enforce specific architectural designs. For
  example, a totally stubby area with one ABR has only one LSA3 for the
  default route, and this option can enforce this. This key is processed
  for any area type. To disable this check, exclude this key.
• max_lsa7: The maximum number of NSSA-external LSAs (LSA type-7) that
  should be present within an area. This inclusive upper bound enforces a
  limit on the number of LSA7 for the purpose of flood reduction and memory
  consumption. It can also enforce specific architectural designs. For
  example, an extranet NSSA acting as a non-transit buffer might be receiving
  a small number of routes from a peer, which can be enforced. This key is
  only process when the area type is "nssa".
  To disable this check, exclude this key.

Device group level

Each device type (ios, iosxr, etc.) has its own group_vars/ file which
contains OS-specific parameters. These should never be changed by consumers
as their main purpose is abstraction, not user input.

• ansible_network_os: A string representing the device OS name. These were
  enumerated in the "Supported Platforms" section earlier in the document.
• commands: A list of strings representing the CLI commands to be
  issued to the device. These collect information from the devices relevant
  to troubleshooting OSPF.

One special consideration is extended to ios. Because classic IOS and
IOS-XE have minor differences in the commands they support, they use
difference command lists. The IOS-related group variables are as follows:

• ios: General IOS parameters, applying to classic and XE variants
• iosclassic: Command list specific to classic IOS devices
• iosxe: Command list specific to IOS-XE devices

An example inventory might look like this:

all:
  children:
    ospf_routers:
      children:
        ios:
          children:
            iosxe:
              hosts:
                CSR1000:
                ISR4451:
            iosclassic:
              hosts:
                C3945E:
                C3750X:

Note that some extra commands are appended to the end of the commands list
which are used for collection only. The output from these commands is written
to the host log which can assist with troubleshooting, but it is not parsed
or checked in any way within the logic of the playbook.

Host level

This playbook aims to minimize the number of host-specific variables as
managing these inventory variables becomes burdensome in large networks.

• my_areas: List of integers representing the areas in which a given router
  participates. For example, a router only in area 0 would use [0]. A
  router in area 0 and 51 would use [0, 51]. Note that the playbook is
  smart enough to identify whether a router is an Area Border Router (ABR)
  or not based on its area membership. The empty list [] is not valid
  since all OSPF routers must belong to at least one area.
  This key is mandantory.
• my_nbr_count: Number of neighbors a specific router is expected to have.
  This is the grand total of all OSPF neighbors in a given process and is
  not checked on a per-interface or per-area basis. It is a positive
  integer. To disable this check, exclude this key. Disabling this check on
  routers with a variable number of neighbors, such as an Internet VPN
  concentrator, could be useful.
• should_be_asbr: Boolean representing whether a router should be an
  Autonomous System Boundary Router (ASBR). A value of true indicates
  that a router should be an ASBR (note that this includes NSSA ABRs)
  and a value of false indicates that a router should not be an ASBR.
  To disable this check, exclude this key.
• should_be_stub_rtr: Boolean representing whether a router should be a
  stub router with max-metric advertised for all links. This reduces the
  likelihood that a router is used for transit. A value of true indicates
  that a router should be a stub router (minor options are not evaluated)
  and a value of false indicates that a router should not be a stub router.
  To disable this check, exclude this key.

Logging

Given the generic nature of the playbook, some tests will fail with generic
error messages. For example, one host may fail because a router had an
incorrect number of actual neighbors, either greater than or less than
the user-configured my_nbr_count expectation. By design, the playbook
lacks granularity to determine which neighbor failed and on which interface.
Logging can be toggled off an on by adjusting the log variable which
can be true or false.

CLI output from all commands is written to a file in the logs/ directory.
A subdirectory for every execution of the playbook is created using
the format nots_<date/time>/ which contains all the individual log files.The date/time uses ISO8601 short format, such as20180522T134558`. Log files
are not version controlled and are excluded from git automatically. An example
log directory after three playbook runs against an inventory of two hosts
(csr1 and csr2), would yield something like this:

$ tree logs/
logs
├── nots_20180522T192916
│   ├── csr1.txt
│   └── csr2.txt
├── nots_20180522T194610
│   ├── csr1.txt
│   └── csr2.txt
└── nots_20180522T197133
    ├── csr1.txt
    └── csr2.txt

The contents of each log file begin with heading and trailing comment blocks
to show the command issued with its output. These logs are useful for finding
out why the playbook failed without having to manually log into failing hosts.
The example below shows the beginning of an IOS-based platform log file with
many redactions for brevity:

$ cat logs/nots_20180522T194610/csr1.txt
!!!
!!! Start command: show ip ospf 1
!!!
Routing Process "ospf 1" with ID 10.0.0.1
 Start time: 00:02:24.532, Time elapsed: 00:48:30.920
 Supports only single TOS(TOS0) routes
[snip, more output]
!!!
!!! End command:   show ip ospf 1
!!!
!!!
!!! Start command: show ip ospf 1 neighbor
!!!
Neighbor ID     Pri   State           Dead Time   Address         Interface
10.0.0.2          1   FULL/DR         00:00:39    192.168.102.2   Tunnel102
10.0.0.2          0   FULL/  -        00:00:37    192.168.101.2   Tunnel101
!!!
!!! End command:   show ip ospf 1 neighbor
!!!
[snip, more commands]

FAQ

Q: Most code across IOS, IOS-XR, and NX-OS is the same. Why not combine it?
A: The goal is to support more platforms in the future such as Cisco
ASA-OS, and possibly non-Cisco devices. These devices will likely return
different sets of information. This tool is designed to be simple,
not particularly advanced through layered abstractions.

Q: Why not use an API like RESTCONF or NETCONF instead of SSH + CLI?
A: This tool is designed for risk-averse users or managers that are not
rapidly migrating to API-based management. It is not an infrastructure-as-code
solution and does not manage device configurations. All of the commands used
in the playbook can be issued at privilege level 1 to further reduce risk.
With the exception of updating the login credentials and populating the
necessary variables, there is no complex setup work required.

Q: Why not parse the OSPF interfaces? Many errors occur at this level.
A: Parsing individual interfaces would require state declarations on a
per-host basis to determine what each interface should have. This defeats
the purpose of a simple, low-effort solution which uses only area and process
level parameters for verification. Furthermore, the detailed statistics
checking will alert the user to many errors (authentication, MTU mismatch, etc)
at a more general level. The user can check the logs to see the exact commands,
which includes the non-parsed interface text.

Q: For NX-OS why didn't you use the | json filter from the CLI?
A: While this would have saved a lot of parsing code, I did not want to
have an inconsistent overall strategy for one network device. Additionally,
the filter does not render milliseconds properly (eg, SPF throttle timers)
which reduced my confidence in its overall accuracy.

Use Case