Hi



Loss access again today from remote client but virtual servers are ok.



Client IP 172.18.2.43



Ping 10.24.7.17 OK Router

Ping 10.24.7.18 FAIL Hypervisor

Ping 10.24.7.19 Virtual server running on service module



C:\>ping 10.24.7.17



Pinging 10.24.7.17 with 32 bytes of data:



Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



Ping statistics for 10.24.7.17:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C

C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Request timed out.



Ping statistics for 10.24.7.18:

Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

Control-C

^C

C:\>ping 10.24.7.19



Pinging 10.24.7.19 with 32 bytes of data:



Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



Ping statistics for 10.24.7.19:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C





C:\>tracert -d 10.24.7.18



Tracing route to 10.24.7.18 over a maximum of 30 hops



1 <1 ms 1 ms 2 ms 172.18.2.33

2 2 ms 2 ms 3 ms 172.18.40.9

3 <1 ms <1 ms <1 ms 172.31.14.39

4 * * ^C



Show Ip interface brief



GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





From virtual Server 10.24.7.19



C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



Ping statistics for 10.24.7.18:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms



From DCUI



# ping 172.18.2.43

PING 172.18.2.43 (172.18.2.43): 56 data bytes



--- 172.18.2.43 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # ping 10.24.7.17

PING 10.24.7.17 (10.24.7.17): 56 data bytes

64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



--- 10.24.7.17 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.270/0.310/0.333 ms



/sbin # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

/sbin # esxcfg-route

VMkernel default gateway is 10.24.7.17

/sbin # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





Are there any other commands, that I can use to see what is going on?





Thanks



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Monday, June 20, 2011 10:51 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

--------------------------------------------------------------
Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

The hypervisor is accessible from the local subnet.

virtual machines running on the hypervisor are not affected

A reload of the hypervisor resolves the problem for a short time but is service affecting.

configuration as below:

interface SM1/0
ip vrf forwarding DISTSCHOOLS
ip address 10.24.7.17 255.255.255.240
service-module ip address 10.24.7.18 255.255.255.240
!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
service-module ip default-gateway 10.24.7.17
hold-queue 60 out
interface SM1/1
no ip address
end

no real timeline when the problem manifests

I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
--
To respond to this post, please click the following link:

<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------

Hi ISMAIL,



Just want to make sure I understand the issue correctly:



1) It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a new VM network port group under vswitch1?

2) When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

3) Do you have any nic teaming enabled on vsiwtch1?





Thanks,

Radhika



From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 1:26 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi



Loss access again today from remote client but virtual servers are ok.



Client IP 172.18.2.43



Ping 10.24.7.17 OK Router

Ping 10.24.7.18 FAIL Hypervisor

Ping 10.24.7.19 Virtual server running on service module



C:\>ping 10.24.7.17



Pinging 10.24.7.17 with 32 bytes of data:



Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



Ping statistics for 10.24.7.17:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C

C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Request timed out.



Ping statistics for 10.24.7.18:

Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

Control-C

^C

C:\>ping 10.24.7.19



Pinging 10.24.7.19 with 32 bytes of data:



Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



Ping statistics for 10.24.7.19:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C





C:\>tracert -d 10.24.7.18



Tracing route to 10.24.7.18 over a maximum of 30 hops



1 <1 ms 1 ms 2 ms 172.18.2.33

2 2 ms 2 ms 3 ms 172.18.40.9

3 <1 ms <1 ms <1 ms 172.31.14.39

4 * * ^C



Show Ip interface brief



GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





From virtual Server 10.24.7.19



C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



Ping statistics for 10.24.7.18:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms



From DCUI



# ping 172.18.2.43

PING 172.18.2.43 (172.18.2.43): 56 data bytes



--- 172.18.2.43 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # ping 10.24.7.17

PING 10.24.7.17 (10.24.7.17): 56 data bytes

64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



--- 10.24.7.17 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.270/0.310/0.333 ms



/sbin # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

/sbin # esxcfg-route

VMkernel default gateway is 10.24.7.17

/sbin # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





Are there any other commands, that I can use to see what is going on?





Thanks



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Monday, June 20, 2011 10:51 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

--------------------------------------------------------------
Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

The hypervisor is accessible from the local subnet.

virtual machines running on the hypervisor are not affected

A reload of the hypervisor resolves the problem for a short time but is service affecting.

configuration as below:

interface SM1/0
ip vrf forwarding DISTSCHOOLS
ip address 10.24.7.17 255.255.255.240
service-module ip address 10.24.7.18 255.255.255.240
!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
service-module ip default-gateway 10.24.7.17
hold-queue 60 out
interface SM1/1
no ip address
end

no real timeline when the problem manifests

I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
--
To respond to this post, please click the following link:

<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
or simply reply to this email.

Hi Radhika



Good to hear from you.



Configuration is as below: ( Full config is in the post relating to vrf )



interface SM1/0

ip vrf forwarding DISTSCHOOLS

ip address 10.24.7.17 255.255.255.240

service-module ip address 10.24.7.18 255.255.255.240

!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V

service-module ip default-gateway 10.24.7.17

hold-queue 60 out

!

interface SM1/1

no ip address



In terms of the questions:



1) I create a virtual switch port group under vswitch0 which I moved my virtual machines to

2) Yes ¿ output below



TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.17



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.24.7.17, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.18



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.24.7.18, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms



3) No teaming enabled on nic1 ¿ it is unused



As per Brett¿s suggestion ¿ I will upgrade the IOS to the recommended version and see if this resolves the issue.



Are there any commands you want me to run before I shut the module down?



Kind regards



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 9:46 PM
To: cdicuser@developer.cisco.com
Subject: New Message from Radhika Miriyala in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



Radhika Miriyala has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi ISMAIL,



Just want to make sure I understand the issue correctly:



1) It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a new VM network port group under vswitch1?

2) When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

3) Do you have any nic teaming enabled on vsiwtch1?





Thanks,

Radhika



From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 1:26 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi



Loss access again today from remote client but virtual servers are ok.



Client IP 172.18.2.43



Ping 10.24.7.17 OK Router

Ping 10.24.7.18 FAIL Hypervisor

Ping 10.24.7.19 Virtual server running on service module



C:\>ping 10.24.7.17



Pinging 10.24.7.17 with 32 bytes of data:



Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



Ping statistics for 10.24.7.17:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C

C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Request timed out.



Ping statistics for 10.24.7.18:

Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

Control-C

^C

C:\>ping 10.24.7.19



Pinging 10.24.7.19 with 32 bytes of data:



Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



Ping statistics for 10.24.7.19:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C





C:\>tracert -d 10.24.7.18



Tracing route to 10.24.7.18 over a maximum of 30 hops



1 <1 ms 1 ms 2 ms 172.18.2.33

2 2 ms 2 ms 3 ms 172.18.40.9

3 <1 ms <1 ms <1 ms 172.31.14.39

4 * * ^C



Show Ip interface brief



GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





From virtual Server 10.24.7.19



C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



Ping statistics for 10.24.7.18:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms



From DCUI



# ping 172.18.2.43

PING 172.18.2.43 (172.18.2.43): 56 data bytes



--- 172.18.2.43 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # ping 10.24.7.17

PING 10.24.7.17 (10.24.7.17): 56 data bytes

64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



--- 10.24.7.17 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.270/0.310/0.333 ms



/sbin # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

/sbin # esxcfg-route

VMkernel default gateway is 10.24.7.17

/sbin # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





Are there any other commands, that I can use to see what is going on?





Thanks



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Monday, June 20, 2011 10:51 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

--------------------------------------------------------------
Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

The hypervisor is accessible from the local subnet.

virtual machines running on the hypervisor are not affected

A reload of the hypervisor resolves the problem for a short time but is service affecting.

configuration as below:

interface SM1/0
ip vrf forwarding DISTSCHOOLS
ip address 10.24.7.17 255.255.255.240
service-module ip address 10.24.7.18 255.255.255.240
!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
service-module ip default-gateway 10.24.7.17
hold-queue 60 out
interface SM1/1
no ip address
end

no real timeline when the problem manifests

I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
--
To respond to this post, please click the following link:

<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
or simply reply to this email.
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098543>
or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------

Please Go ahead and upgrade the IOS image as suggested by Brett. If you still see the issue after the IOS upgrade we can discuss further steps to debug.



From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 2:15 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi Radhika



Good to hear from you.



Configuration is as below: ( Full config is in the post relating to vrf )



interface SM1/0

ip vrf forwarding DISTSCHOOLS

ip address 10.24.7.17 255.255.255.240

service-module ip address 10.24.7.18 255.255.255.240

!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V

service-module ip default-gateway 10.24.7.17

hold-queue 60 out

!

interface SM1/1

no ip address



In terms of the questions:



1) I create a virtual switch port group under vswitch0 which I moved my virtual machines to

2) Yes ¿ output below



TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.17



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.24.7.17, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.18



Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.24.7.18, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms



3) No teaming enabled on nic1 ¿ it is unused



As per Brett¿s suggestion ¿ I will upgrade the IOS to the recommended version and see if this resolves the issue.



Are there any commands you want me to run before I shut the module down?



Kind regards



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 9:46 PM
To: cdicuser@developer.cisco.com
Subject: New Message from Radhika Miriyala in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



Radhika Miriyala has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi ISMAIL,



Just want to make sure I understand the issue correctly:



1) It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a new VM network port group under vswitch1?

2) When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

3) Do you have any nic teaming enabled on vsiwtch1?





Thanks,

Radhika



From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Tuesday, June 21, 2011 1:26 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
--------------------------------------------------------------
Hi



Loss access again today from remote client but virtual servers are ok.



Client IP 172.18.2.43



Ping 10.24.7.17 OK Router

Ping 10.24.7.18 FAIL Hypervisor

Ping 10.24.7.19 Virtual server running on service module



C:\>ping 10.24.7.17



Pinging 10.24.7.17 with 32 bytes of data:



Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



Ping statistics for 10.24.7.17:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C

C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Request timed out.



Ping statistics for 10.24.7.18:

Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

Control-C

^C

C:\>ping 10.24.7.19



Pinging 10.24.7.19 with 32 bytes of data:



Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



Ping statistics for 10.24.7.19:

Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms

Control-C

^C





C:\>tracert -d 10.24.7.18



Tracing route to 10.24.7.18 over a maximum of 30 hops



1 <1 ms 1 ms 2 ms 172.18.2.33

2 2 ms 2 ms 3 ms 172.18.40.9

3 <1 ms <1 ms <1 ms 172.31.14.39

4 * * ^C



Show Ip interface brief



GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





From virtual Server 10.24.7.19



C:\>ping 10.24.7.18



Pinging 10.24.7.18 with 32 bytes of data:



Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



Ping statistics for 10.24.7.18:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms



From DCUI



# ping 172.18.2.43

PING 172.18.2.43 (172.18.2.43): 56 data bytes



--- 172.18.2.43 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # ping 10.24.7.17

PING 10.24.7.17 (10.24.7.17): 56 data bytes

64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



--- 10.24.7.17 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.270/0.310/0.333 ms



/sbin # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

/sbin # esxcfg-route

VMkernel default gateway is 10.24.7.17

/sbin # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





Are there any other commands, that I can use to see what is going on?





Thanks



Ismail







From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
Sent: Monday, June 20, 2011 10:51 PM
To: cdicuser@developer.cisco.com
Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

--------------------------------------------------------------
Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

The hypervisor is accessible from the local subnet.

virtual machines running on the hypervisor are not affected

A reload of the hypervisor resolves the problem for a short time but is service affecting.

configuration as below:

interface SM1/0
ip vrf forwarding DISTSCHOOLS
ip address 10.24.7.17 255.255.255.240
service-module ip address 10.24.7.18 255.255.255.240
!Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
service-module ip default-gateway 10.24.7.17
hold-queue 60 out
interface SM1/1
no ip address
end

no real timeline when the problem manifests

I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
--
To respond to this post, please click the following link:

<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
or simply reply to this email.
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098543>
or simply reply to this email.



---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------
--
To respond to this post, please click the following link:
<http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4096956>
or simply reply to this email.

Hi Ismail,

Please send us the data below so that we can begin to do an analysis of what's happening on your system.

1. From the vSphere Client, please export the 'System Logs' and send over.

2. On the router before and after the failure occurs please type 'show ip route' and send us the results. We want to see if the routes have changed compared to when working and when broken.

3. Please make sure the router is printing logs to the terminal, and send us the captured logs before/after the failure occurs.

Thanks,

Brett

Another one that did not get through



From: Ismail Patel
Sent: Thursday, June 23, 2011 12:44 AM
To: cdicuser@developer.cisco.com
Cc: Brett Tiller (brtiller) (brtiller@cisco.com)
Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



Unexpected service module restart ¿ I restarted the management interface to see if this restored connectivity which it did not. Left it to do some work and came back to find my virtual servers had rebooted.



From virtual windows server log



The previous system shutdown at 00:15:38 on 23/06/2011 was unexpected.



For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.







Nothing in router log:



*Jun 22 23:27:16.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

e to down

*Jun 22 23:27:19.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

e to up

*Jun 22 23:27:20.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth

ernet0/2, changed state to up

*Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_

MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY



Service module on-line from remote client



TESTSCHRTR101742#show ip route vrf DISTSCHOOLS



Routing Table: DISTSCHOOLS

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override



Gateway of last resort is 172.31.14.1 to network 0.0.0.0



S* 0.0.0.0/0 [1/0] via 172.31.14.1

10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks

C 10.24.0.0/22 is directly connected, Vlan2

L 10.24.0.1/32 is directly connected, Vlan2

C 10.24.7.1/32 is directly connected, Loopback0

C 10.24.7.16/28 is directly connected, SM1/0

L 10.24.7.17/32 is directly connected, SM1/0

172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks

C 172.31.14.0/24 is directly connected, GigabitEthernet0/0.211

L 172.31.14.39/32 is directly connected, GigabitEthernet0/0.211



Attached are the latest logs from server



Many thanks



Ismail









---------------------------------------------------------------------
DISCLAIMER: This email and files transmitted are
confidential and are intended solely for the use of the
intended recipient. If you are not the intended
recipient, or the person responsible for delivering it to
the intended recipient, you may not copy, disclose,
distribute or use it in any unauthorised manner. If you
have received this email in error please notify us by
email to postmaster@wolverhampton.gov.uk and then delete
it and any attachments accompanying it. Please note that
Wolverhampton City Council cannot guarantee that this
message or any attachments are virus free or have not been
intercepted and amended.
Any views or opinions expressed within this email are
those of the author and may not necessarily reflect those
of Wolverhampton City Council and no contractual
arrangement is intended to arise from this communication.
---------------------------------------------------------------------

HI Ismail,

Please upload the files via the steps below and let us know the filenames as well.

1. Connect to ftp.cisco.com via anonymous ftp.
2. Files must be uploaded into a folder called 'incoming', which is not visible to anonymous users
3. If you are using a text-based ftp client, type the command "cd incoming"
4. If you are using a GUI ftp client such as FileZilla, enter 'incoming' manually as the destination for the remote site
Upload the file as normal. You will not be able to see it on the server, as commands to list the files in that folder have been disabled. You may see error messages related to that.

NOTE: You cannot upload a file with the same name more than once as this creates the potential for other users to overwrite the files you have uploaded. If you need to re-upload the same file, either use a new filename or ask the Cisco employee you are working with to remove the file from the server.

Thanks,

Brett

Hi Guys,

1) Files uploaded

IEP-10.24.7.18-vmsupport-2011-06-23@00-34-47.tgz
IEP-WCC-14CA5C74E84-viclient-support-2011-06-23@00-35-11.zip

2) Reset of management network was done from the console from the menu option

3) Show ip route command output from working module is in the above post when after the module rebooted.

The previous post had the output from when it was not accessible from remote network.

4) show log below:

*Jun 21 21:49:12.459: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Jun 21 21:49:12.459: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Jun 21 21:49:12.459: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Jun 21 21:49:12.459: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Jun 21 21:49:12.512: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.130.2.2 p
ort 514 started - CLI initiated
*Jun 21 21:49:13.180: %LINK-3-UPDOWN: Interface FastEthernet0/0/3, changed state
to up
*Jun 21 21:49:41.668: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2, cha
nged state to up
*Jun 21 21:49:56.480: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
MODULE_STATE_WREG to SERVICE_MODULE_STATE_STDY
*Jun 22 23:11:37.713: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
MODULE_STATE_STDY to SERVICE_MODULE_STATE_WREG
*Jun 22 23:24:39.055: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
MODULE_STATE_WREG to SERVICE_MODULE_STATE_ERRQ
*Jun 22 23:24:40.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to down
*Jun 22 23:24:41.523: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to down
*Jun 22 23:24:43.531: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to up
*Jun 22 23:24:44.531: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to up
*Jun 22 23:24:53.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to down
*Jun 22 23:24:54.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to down
*Jun 22 23:24:57.532: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to up
*Jun 22 23:24:58.532: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to up
*Jun 22 23:25:54.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to down
*Jun 22 23:25:55.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to down
*Jun 22 23:25:58.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to up
*Jun 22 23:25:59.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to up
*Jun 22 23:26:24.576: %SM_INSTALL-6-INST_RBIP: SM1/0 received msg: RBIP Registra
tion Request
*Jun 22 23:26:25.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to down
*Jun 22 23:26:26.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to down
*Jun 22 23:26:33.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to up
*Jun 22 23:26:34.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to up
*Jun 22 23:27:15.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to down
*Jun 22 23:27:16.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to down
*Jun 22 23:27:19.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
e to up
*Jun 22 23:27:20.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
ernet0/2, changed state to up
*Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY

Ismail

Hi Ismail,

Based upon the router logs you've provided above, it appears that the gig0/2 interface is flapping. Are you seeing this up/down status on the interface constantly? If so, were you seeing this same status when you were using IOS 15.1-3T?

Thanks,

Brett

Hi

I believe the issue is related to CEF and VRF-lite

CEF table when service module is accessible from remote networks ( service module is listed )..
When the module is not accessible then entry for the service module is missing. Working table below

TESTSCHRTR101742#sh ip cef vrf DISTSCHOOLS
Prefix Next Hop Interface
0.0.0.0/0 172.31.14.1 GigabitEthernet0/0.211
0.0.0.0/8 drop
0.0.0.0/32 receive
10.24.0.0/22 attached Vlan2
10.24.0.0/32 receive Vlan2
10.24.0.1/32 receive Vlan2
10.24.0.10/32 attached Vlan2
10.24.0.11/32 attached Vlan2
10.24.3.255/32 receive Vlan2
10.24.7.1/32 receive Loopback0
10.24.7.16/28 attached SM1/0
10.24.7.16/32 receive SM1/0
10.24.7.17/32 receive SM1/0
10.24.7.18/32 attached SM1/0
10.24.7.19/32 attached SM1/0
10.24.7.20/32 attached SM1/0
10.24.7.31/32 receive SM1/0

regards

Ismail