Netdevice
A netdevice (netdev) is a Linux kernel construct which represents a networking element. It can represent a physical interface like a front-end switch port, or a logical interface such as a tunnel. The netdev files on NX-OS exist under /proc/net/dev filesystem. The names for netdevices are similar to the NX-OS interface names. For example, Ethernet1/1 in NX-OS (port 1 in slot 1) refers to the corresponding Linux interface name of Eth1-1. It is important to note that interface names within Linux are limited to 15 characters, therefore ‘Ethernet’ is shortened to ‘Eth’. This is consistent with the naming in show interface brief within NX-OS.
Using ifconfig on a Nexus Switch
Linux network utilities, commonly used by server admins, can now be used to configure, monitor, troubleshoot and manage the switch.
Using ifconfig to view interfaces:
root@n9k-sw-1# ifconfig -a | grep Eth
Eth1-1 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:98
Eth1-2 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:99
Eth1-3 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:9a
Eth1-4 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:9b
Eth1-5 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:9c
Using ifconfig to enable/disable an interface and verify in NX-OS
root@n9k-sw-1# ifconfig Eth2-4 down
root@n9k-sw-1# vsh -c "show interface Eth2/4"
Ethernet2/4 is down (Administratively down)
admin state is down, Dedicated Interface
Hardware: 10000/40000 Ethernet, address: 6412.25ed.787f (bia 6412.25ed.787f)
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec
Notice above vsh was used to run a NX-OS CLI command. When working within the Bash Shell, the vsh command can be used to interact with the NX-OS parser.
Assigning a Layer 3 Address to an Interface in Linux
To maintain consistency between NX-OS and Linux, there are validation checks that are included for several operations. As an example, if you attempt to assign a Layer 3 address to a Layer 2 interface in Linux or via a third-party application, an error will be presented.
Error Addressing Layer 2 Interface
root@n9k-sw-1# vsh -c "show run int Eth2/4"
interface Ethernet2/4
switchport
root@n9k-sw-1# ifconfig Eth2-4 192.168.2.2 netmask 255.255.255.0
SIOCSIFADDR: Cannot assign requested address
SIOCSIFNETMASK: Cannot assign requested address
If you change the interface mode to Layer 3 within NX-OS you can now assign a Layer-3 address in Linux:
Assign a Layer 3 address
root@n9k-sw-1# ip netns exec default bash
root@n9k-sw-1# vsh -c "config terminal ; interface Eth2/4 ; no switchport ; no shutdown"
l root@n9k-sw-1# ifconfig Eth2-4 192.168.2.2 netmask 255.255.255.0
root@n9k-sw-1# ifconfig Eth2-4
Eth2-4 Link encap:Ethernet HWaddr 10:05:ca:f5:ee:97
inet addr:192.168.2.2 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1837767 errors:0 dropped:1837763 overruns:0 frame:0
TX packets:70576 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:95625447 (91.1 MiB) TX bytes:17898072 (17.0 MiB)
root@n9k-sw-1# vsh -c "show run int Eth2/4"
interface Ethernet2/4
no switchport
ip address 192.168.2.2/24
no shutdown
Using ethtool on a Nexus Switch
Ethtool is a useful utility to view driver level interface statistics. This can be used to get information about front panel interfaces. An example is shown here, which gathers port statistics for the Ethnernet2/4 interface.
Interface port statistics:
root@n9k-sw-1# ethtool -S Eth2-4
NIC statistics:
speed: 40000
port_delay: 10
port_bandwidth: 40000000
admin_status: 1
oper_status: 1
port_mode: 0
reset_counter: 6
load-interval-1: 30
rx_bit_rate1: 256
rx_pkt_rate1: 0
tx_bit_rate1: 144
tx_pkt_rate1: 0
load-interval-2: 300
rx_bit_rate2: 248
Using tcpdump on a Nexus Switch
Tcpdump is a packet analyzer utility which can be run directly from the command line in the default namespace for a front panel interface. The example below uses tcpdump to monitor packets on an interface and store them in a pcap (packet capture) file named file.pcap. Use the -i flag to specify an interface.
tcpdump packet capture:
root@n9k-sw-1# tcpdump -i Eth2-4 -w /bootflash/file.pcap
tcpdump: WARNING: Eth2-4: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on Eth2-4, link-type EN10MB (Ethernet), capture size 65535 bytes
23:06:54.402365 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8001.84:b8:02:0e:f8:3b.8031, length 43
23:06:56.402207 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8001.84:b8:02:0e:f8:3b.8031, length 43
Using route commands on a Nexus Switch
Managing Routing Within Linux
Routing can be configured within Open NX-OS Linux. Static routes can be added and deleted directly using the route command. Any routing changes made will be immediately reflected within the NX-OS routing tables.
Set an IP address on e1/1:
n9k-sw-1(config)# int eth2/4
n9k-sw-1(config-if)# ip address 192.168.1.2/24
Ensure you are root user and in default namespace to configure frontpanel interfaces:
n93k-1-pm# run bash
bash-4.2$ id
uid=2002(admin) gid=503(network-admin) groups=503(network-admin)
bash-4.2$ su - root
Password:
root@n93k-1-pm#ip netns exec default bash
The following routing changes are made completely in bash.
Display routes
root@n9k-sw-1# route
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 Eth2-4
127.1.0.0 * 255.255.0.0 U 0 0 0 veobc
127.1.2.0 * 255.255.255.0 U 0 0 0 veobc
Add a route
root@n9k-sw-1# ip route add 192.168.3.0/24 via 192.168.1.2 dev Eth2-4
Display updated route table:
root@n9k-sw-1# route
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 Eth2-4
192.168.3.0 192.168.1.2 255.255.255.0 UG 0 0 0 Eth2-4
127.1.0.0 * 255.255.0.0 U 0 0 0 veobc
127.1.2.0 * 255.255.255.0 U 0 0 0 veob
Verify in NX-OS:
n9k-sw-1# sh run | i "ip route" ip route 192.168.2.0/24 Ethernet1/192.168.1.2 1
n9k-sw-1# sh ip route
192.168.1.0/24, ubest/mbest: 1/0, attached *via 192.168.1.1, Eth1/1, [0/0], 00:02:08, direct
192.168.1.2/32, ubest/mbest: 1/0, attached *via 192.168.1.1, Eth1/1, [0/0], 00:02:08, local
192.168.2.0/24, ubest/mbest: 1/0 *via 192.168.1.2, Eth1/1, [1/0], 00:01:37, static
n9k-sw-1#
Remove a route:
n9k-sw-1# ip route del 192.168.3.0/24 via 192.168.1.2 dev Eth2-4
Display updated route table:
bash-4.2# route
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 Eth1-1
127.1.0.0 * 255.255.0.0 U 0 0 0 veobc
127.1.2.0 * 255.255.255.0 U 0 0 0 veobc
bash-4.2#