Wireless Troubleshooting Tools

Wireless Detector

Introduction

Wireless Clients may fail randomly, making troubleshooting and detection difficult, especially on large scenarios. Sometimes, it is not possible to know in advance which client will fail, additionally client failures may be transient and hard to reproduce

The Wireless Detector tool, leverages the Always On Tracing feature from Catalyst 9800 IOS-XE controllers, to pull client delete reasons, classify them, and generate reports on what happed. No additional debugging is required.

The tool can export tracing information per client, grouping events sharing similar time windows for simplicity, additionally, you can set it up to generate email reports with event summaries

Default data collection times are set for "safe" operation on large scale scenarios. It is advisable to change client counts and/or period to adapt to your needs. For example, changing from 10 minutes to 1 hour, or 1 day

Features

  • Works on any 9800 controller model. Tested on 17.3/17.9

  • Collect information on client failures from Client Orchestrator Delete Events

  • Classify per delete reason type

  • Filter by severity:

    • Normal events

    • May happen but could need validation

    • Errors either client or network side

  • Send report by email, or display in console

  • Limit clients/events tracked

  • Export Always on tracing, grouping events and clients

  • Set time range for data collection

  • EEM scripts runs on any 9800/version

  • Defaults:

    • Time to scan: 10 minutes

    • Clients tracked: 10

    • Time stamps per delete type: 10

    • Email report: disabled

  • Around 180 different delete types for 17.3, and 320+ for 17.9

Install Steps

  • Create directory to store script + supporting files

9800-l#mkdir eem_lib

  • Change into directory

9800-l# cd eem_lib

  • Copy file to location

9800-l#copy ftp://SERVERIP/wireless_detector.tcl wireless_detector.tcl

  • Configure script directory in config mode:

9800-cl#config Configuring from terminal, memory, or network [terminal]? Enter configuration commands, one per line. End with CNTL/Z. 9800-cl(config)# event manager directory user policy bootflash:/eem_lib

  • Register script in config mode:

9800-cl#config Configuring from terminal, memory, or network [terminal]? Enter configuration commands, one per line. End with CNTL/Z. 9800-cl(config)# event manager policy wireless_detector.tcl type user

Configuration

  • By default, the script will run with "safe" defaults, generating a report to the active CLI session. If needed, you can modify options through environmental values configured in your controller.
  • Please check Email Configuration for details on how to setup email reporting
  • These parameters are set from config mode in controller

! 1 for errors, 2 for warnings, 3 for informational. 1 will include all levels event manager environment wd_minimum_level 2

! Format has to be number plus one word out of minutes or hours or days event manager environment wd_period "10 minutes"

! How many clients to report in detail, by default is 10 event manager environment wd_clients_detail_count 10

! How many event time stamps to track per client event manager environment wd_max_time_stamps 10

! Set to 1, if you want to collect Always On traces for the events detected per client, 0 will disable log collection (default) event manager environment wd_log_collection 0

! How many files to collect per client, if wd_log_collection is set to 1
event manager environment wd_max_files_run 3

! Events are combined in time windows, if 2 events have overlapping time, they will be set in same file. This is how many minutes to start the time window from the event time stamp
event manager environment wd_client_logs_start 5

! This sets how long to collect logs, after the event time stamp event manager environment wd_client_logs_stop 5

! Location to store log files. This should match the command: "event manager directory user policy "bootflash:/eem_lib"
event manager environment wd_directory bootflash:/eem_lib

Report Generation Mode.

By default, report is displayed on CLI only, and not stored. The tool supports exporting the results to a report file, suppressing reporting to CLI

  • Mode 1: set variable to 1 to store report in HTML format (recommended)
  • Mode 2: set variable to 2 to store report in text format
  • Disabled: set variable to 0, report is either sent by email, or displayed in CLI (default)

If report generation is enabled, it will be stored on the configured directory as "report_{timestamp}.{extension}" for example report_20230524T12:31:00.html

event manager environment wd_report 0

Executing the Script

  • By default, it is set to execute on demand:

9800-l#event manager run wireless_detector.tcl

Wireless Client Problem Detector v.2

Preparing application...

Collecting client delete data, this may take a while...

Avoid running the script twice, you can use the command "show event manager policy active" to confirm if it has finished

Processing data...

Sending email results...

E-mail sent!

Updating the script

  • You can use the event update CLI to install a new version.

  • Rarely, this command may fail, workaround is to close CLI and try again

  • From enable mode:

9800-l#event manager update user policy name wireless_detector.tcl repository ftp://yourftpserver

Example output:

%EEM: Update will use the repository path: ftp://192.168.0.99:2121

%EEM: Attempting to copy ftp://192.168.0.99:2121/wireless_detector.tcl to bootflash:/eem_lib/wireless_detector.tcl!

[OK - 40658/4096 bytes]

%EEM: Copied 40658 bytes from ftp://192.168.0.99:2121/wireless_detector.tcl to bootflash:/eem_lib/wireless_detector.tcl

%EEM: Policy wireless_detector.tcl has been successfully copied and re-registered

Email Report Configuration

  • By default, the script will only generate text report to the active CLI. To use the email report feature, you would need a SMTP relay server for your domain, reachable from the controller management IP address.

  • Important:: ensure your controller can do DNS resolution, and has access to email server on port 25, otherwise email can't be sent

  • Set wd_send_email variable to 1 to send email report, and 0 for console report only

  • From config mode:

9800cl-1(config)#event manager environment wd_send_email 1

9800cl-1(config)#event manager environment wd_email_server yourmailserver.exampledomain.com

9800cl-1(config)#event manager environment wd_email_from yoursourceaddress@exampledomain.com

9800cl-1(config)#event manager environment wd_email_to yourdestination@exampledomain.com

  • Optional (depending your setup). Configure DNS resolution:

9800cl-1(config)# ip domain name exampledomain.com ! Important: replace with your DNS servers 9800cl-1(config)# ip name-server 208.67.222.123 208.67.220.123

  • Optional, validate if controller can resolve server name, and reach mail port:

9800-cl#telnet yourmailserver.exampledomain.com 25 Trying yourmailserver.exampledomain.com (x.x.x.x, 25)... Open 220 ****************************

Email Reports

Wireless Detector can send email reports through SMTP server reachable from your controller Requisites:

  • DNS resolution working, you should add your DNS servers and domain. From config mode:

    ip domain name yourdomain.com ip domain lookup ! this is the default unless disabled ip name-server 208.67.222.123 208.67.220.123 ! replace with your server address

  • SMTP server should allow email relay for the source/destination accounts, and controller should be able to reach it on port 25

  • IOS-XE supports SMTP authentication, setting the server with the format:

    event manager environment wd_email_server user:password@server.address.com

  • Currently, it does not support TLS or usernames that include domain name or @ character when using EEM with TCL scripts

  • Email report will contain summary of the events per type, plus detailed information on each client mac address, times each delete event is seen, timestamps, and links to the explanation for the delete reason

Example Config

  • Configuration to add in controller, to track medium level events (warning) for 10 clients, collect logs for 1 day, and send results by email, no local report This would be for a controller with low load

event manager environment wd_email_from example@exampledomain.com event manager environment wd_email_to admin@exampledomain.com event manager environment wd_email_server server.exampledomain.com event manager environment wd_clients_detail_count 10 event manager environment wd_max_time_stamps 10 event manager environment wd_max_files_run 3 event manager environment wd_client_logs_start 5 event manager environment wd_directory bootflash:/eem_lib event manager environment wd_client_logs_stop 5 event manager environment wd_log_collection 0 event manager environment wd_period "1 day" event manager environment wd_send_email 1 event manager environment wd_report 0 event manager environment wd_minimum_level 2

  • Example with report stored locally, in HTML, no email sent, review last 1 hour of logs, and track 20 clients

event manager environment wd_clients_detail_count 20 event manager environment wd_max_time_stamps 10 event manager environment wd_max_files_run 3 event manager environment wd_client_logs_start 5 event manager environment wd_directory bootflash:/eem_lib event manager environment wd_client_logs_stop 5 event manager environment wd_log_collection 0 event manager environment wd_period "1 hour" event manager environment wd_send_email 0 event manager environment wd_report 1 event manager environment wd_minimum_level 1

Next