For the purpose of illustrating this particular use-case, we'll use two Nexus devices connected back to back.

1. Install Splunk forwarder as RPM on NX-OS

  n9k-sw-1# guestshell  
  bash-4.2# yum install splunkforwarder-6.2.3-264376-linux-2.6-x86_64.rpm 

2. Start Splunk

  bash-4.2# splunk start \--accept-license  

3. Add forward server to push data to Splunk Enterprise

  bash-4.2# splunk add forward-server <splunk enterprise>:9997  

4. Enable forwarder

  bash-4.2# splunk restart  


5) Configure Splunk forwarder  

Lastly, configure the Splunk forwarder to watch a monitor file for incoming data.  When any data is written to the monitor file, the Splunk forwarder will pick that data up and send it to the Splunk server. 

```shell
  bash-4.2#splunk add monitor /bootflash/home/admin/monitor_file

The Splunk forwarder can be checked to see what files it is monitoring:

  bash-4.2#splunk list monitor  

Sample scripts to forward data to the Splunk enterprise. All scripts that will be run by Splunk Forwarder should be placed in $SPLUNK_HOME/bin/scripts folder.

6. Create the interface-counter.py script

In addition to capturing IP SLA data, other data sources can be captured as well and sent to the Splunk Collector. As an example, the script below will gather interface counter data which is picked up by the Splunk Universal forwarder. Using the following script interfaces with NX-API CLI functionality to grab interface data through a JSON-RPC call, the resulting data is stored in the response dictionary.

The script used to push any statistics can be easily developed using NX-API CLI. (nxapi URI)
Use the nxapi sandbox to cut-and-paste the python script with the desired 'show command'. (In the sandbox select output format as JSON, and in the request pane select python).

(sample generated script from the sandbox for 'show interface counter' CLI)

#!/usr/bin/python  
  
  import os, json, sys, requests  
  
  Modify these please  
  
  
  url='https://<SWITCH_MGMT_IP>/ins'  
  switchuser='<USERNAME>'  
  switchpassword='<PASSWORD>'  
  
  myheaders={'content-type':'application/json'}  
  payload={  
    "ins_api": {  
      "version": "1.0",  
      "type": "cli_show",  
      "chunk": "0",  
      "sid": "1",  
      "input": "show interface ethernet1/13",  
      "output_format": "json"  
    }  
  }  
  
  response = requests.post(url,data=json.dumps(payload), headers=myheaders,auth=(switchuser,switchpassword)).json()  
  
  #Add just this line.to push/forward json output to splunk server.  
  print json.dumps(response)  

7. Create data-forwarder.sh script

This is a wrapper shell script that will execute specific python scripts.

  `#!/bin/bash`  
  unset LD_LIBRARY_PATH  
  /opt/splunkforwarder/bin/scripts/interface-counter.py  

Configure the inputs.conf file according to the sample shown below:

  script://$SPLUNK_HOME/bin/scripts/data-forwarder.sh  
  interval = 60  
  sourcetype = json  

The script listed above can be modified to write the response dictionary data directly to the monitor_file which the Splunk universal forwarder will then pick up and send to the collector.

8. Executing the Splunk forwarder

For the IP SLA example, data specific to IP SLA can also be sent to the Forwarder. Here is an example where jitter data is gathered and sent to the forwarder.

Splunk search string and graph

Search string to plot the average max jitter time from IP SLA sender to responder every minute:

  host=lp1 sourcetype=json "udp-jitter” -> Filter the udp-jitter json formatted event from the host  
  ins_api.outputs.output.body.TABLE_common.ROW_common.latest-return-code="1” -> Filter successful event  
  earliest=-1m -> Filter events over the last one minute  
  | stats avg(ins_api.outputs.output.body.TABLE_jitter{}.ROW_jitter.sd-jitter-max) as avg_max_jitter_time -> Calculate the average of max-jitter-time from source to destination  
  | gauge avg_max_jitter_time 20010 20025 20040 20055 -> Plot the value on a gauge