Thursday, November 06, 2014

ESXi Network Troubleshooting

Introduction

As VMware Expert I had a chance and privilege to use VMware Learning Zone. There are excellent training videos. Today I would like to blog about useful commands trained on video training “Network Troubleshooting at the ESXi Command Line”.  If you ask me I have to say that Vmware Learning Zone has very valuable content and it come really handy during real troubleshooting. 

NIC Adapters Information

To see Network Interface Cards Information you can run following command
~ # /usr/lib/vmware/vm-support/bin/nicinfo.sh | more
Network Interface Cards Information.

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address        MTU   Description
----------------------------------------------------------------------------------------
vmnic0  0000:001:00.0  bnx2    Up     1000  Full    14:fe:b5:7d:8d:05  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX
vmnic1  0000:001:00.1  bnx2    Up     1000  Full    14:fe:b5:7d:8d:07  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX
vmnic2  0000:002:00.0  bnx2    Up     1000  Full    14:fe:b5:7d:8d:6d  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX
vmnic3  0000:002:00.1  bnx2    Up     1000  Full    14:fe:b5:7d:8d:6f  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX

NIC:  vmnic0

NICInfo:
   Advertised Auto Negotiation: true
   Advertised Link Modes: 1000baseT/Full, 2500baseT/Full
   Auto Negotiation: true
   Cable Type: FIBRE
   Current Message Level: -1
   Driver Info:
      NICDriverInfo:
         Bus Info: 0000:01:00.0
         Driver: bnx2
         Firmware Version: 7.8.53 bc 7.4.0 NCSI 2.0.13
         Version: 2.2.3t.v55.7
   Link Detected: true
   Link Status: Up
   Name: vmnic0
   PHY Address: 2
   Pause Autonegotiate: false
   Pause RX: true
   Pause TX: true
   Supported Ports: TP, FIBRE
   Supports Auto Negotiation: true
   Supports Pause: true
   Supports Wakeon: true
   Transceiver: internal
   Wakeon: MagicPacket(tm)
Ring parameters for vmnic0:
Pre-set maximums:
RX:             4080
RX Mini:        0
RX Jumbo:       16320
TX:             255
Current hardware settings:
RX:             255
RX Mini:        0
RX Jumbo:       0
TX:             255

Output above is sniped just for vmnic0. You can see useful information like PCI Device ID, Driver, Link Status, Speed, Duplex and MTU for each vmnic.
It also shows detail driver information, FlowControl (Pause Frame) status, cable type. etc.
To find particular vmnic PCI Vendor ID's use command vmkchdev
~ # vmkchdev -l | grep vmnic0
0000:01:00.0 14e4:163a 1028:02dc vmkernel vmnic0

PCI Slot: 0000:01:00.0
VID (Vendor ID): 14e4
DID (Device ID): 163a
SVID (Sub-Vendor ID): 1028
SSID (Sub-Device ID): 02dc

You can use PCI devices Vendor ID’s  to find the latest drivers at VMware Compatibility Guide (http://www.vmware.com/go/hcl/).


Below is another command how to find full details of all PCI devices.
esxcli hardware pci list
If you are interested just for particular vmnic PCI details command below can be used.
~ # esxcli hardware pci list | grep -B 6 -A 29 vmnic0
000:001:00.0
   Address: 000:001:00.0
   Segment: 0x0000
   Bus: 0x01
   Slot: 0x00
   Function: 0x00
   VMkernel Name: vmnic0
   Vendor Name: Broadcom Corporation
   Device Name: Broadcom NetXtreme II BCM5709S 1000Base-SX
   Configured Owner: Unknown
   Current Owner: VMkernel
   Vendor ID: 0x14e4
   Device ID: 0x163a
   SubVendor ID: 0x1028
   SubDevice ID: 0x02dc
   Device Class: 0x0200
   Device Class Name: Ethernet controller
   Programming Interface: 0x00
   Revision ID: 0x20
   Interrupt Line: 0x0f
   IRQ: 15
   Interrupt Vector: 0x2b
   PCI Pin: 0x75
   Spawned Bus: 0x00
   Flags: 0x0201
   Module ID: 4125
   Module Name: bnx2
   Chassis: 0
   Physical Slot: 0
   Slot Description: Embedded NIC 1
   Passthru Capable: true
   Parent Device: PCI 0:0:1:0
   Dependent Device: PCI 0:0:1:0
   Reset Method: Link reset
   FPT Sharable: true

Note: same command can be used for HBA cards by substituting vmnic0 by vmhba0

VLAN Sniffing

The commands below enable VLAN statistics collection on particular vmnic which can be shown and used for troubleshooting.  
esxcli network nic vlan stats set --enabled=true -n vmnic0
~ # esxcli network nic vlan stats get -n vmnic0
VLAN 0
   Packets received: 22
   Packets sent: 0

VLAN 22
   Packets received: 21
   Packets sent: 10

VLAN 201
   Packets received: 28
   Packets sent: 0

VLAN 202
   Packets received: 28
   Packets sent: 0

VLAN 204
   Packets received: 5
   Packets sent: 0

VLAN 205
   Packets received: 5
   Packets sent: 0

Don’t forget to disable VLAN statistics after troubleshooting.

esxcli network nic vlan stats set --enabled=false -n vmnic0


VMkernel Arp Cache

To work with ESXi ARP cache you can use command
esxcli network ip neighbor  
Below is example how to list ARP entries …
~ # esxcli network ip neighbor list
Neighbor   Mac Address        Vmknic   Expiry  State  Type
---------  -----------------  ------  -------  -----  -------
10.2.22.1  5c:26:0a:ae:5a:c6  vmk0    933 sec         Unknown

You can see there just default gateway 10.2.22.1
Let’s ping some other device in the same broadcast domain and look at ARP entries again.
~ # ping 10.2.22.51
PING 10.2.22.51 (10.2.22.51): 56 data bytes
64 bytes from 10.2.22.51: icmp_seq=0 ttl=128 time=0.802 ms

~ # esxcli network ip neighbor list
Neighbor    Mac Address        Vmknic    Expiry  State  Type
----------  -----------------  ------  --------  -----  -------
10.2.22.51  00:0c:29:4a:5b:ba  vmk0    1195 sec         Unknown
10.2.22.1   5c:26:0a:ae:5a:c6  vmk0     878 sec         Unknown

Now you can see  entry for device 10.2.22.51 in ARP table as well. Below is another command to remove ARP entry from ARP table.
~ # esxcli network ip neighbor remove -v 4 -a 10.2.22.51
… and let’s check if ARP entry has been removed.
~ # esxcli network ip neighbor list
Neighbor   Mac Address        Vmknic   Expiry  State  Type
---------  -----------------  ------  -------  -----  -------
10.2.22.1  5c:26:0a:ae:5a:c6  vmk0    817 sec         Unknown

Note: ESXi ARP timeout is 1200 second therefore remove command can be handy in some situations.

VMkernel Routing

Since vSphere 5.1 it is possible to have more than one networking stack. Normally you work with default networking stack.
To show ESXi routing table you can use command
esxcli network ip route ipv4 list  
~ # esxcli network ip route ipv4 list
Network    Netmask        Gateway    Interface  Source
---------  -------------  ---------  ---------  ------
default    0.0.0.0        10.2.22.1  vmk0       MANUAL
10.2.22.0  255.255.255.0  0.0.0.0    vmk0       MANUAL

You can see default gateway 10.2.22.1 used for default networking stack.
Command esxcli network ip connection list shows all IP network connections from and to ESXi host.
~ # esxcli network ip connection list
Proto  Recv Q  Send Q  Local Address                    Foreign Address     State        World ID  CC Algo  World Name
-----  ------  ------  -------------------------------  ------------------  -----------  --------  -------  ---------------
tcp         0       0  127.0.0.1:8307                   127.0.0.1:54854     ESTABLISHED     34376  newreno  hostd-worker
tcp         0       0  127.0.0.1:54854                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work
tcp         0       0  127.0.0.1:443                    127.0.0.1:54632     ESTABLISHED    570032  newreno  rhttpproxy-work
tcp         0       0  127.0.0.1:54632                  127.0.0.1:443       ESTABLISHED   1495503  newreno  python
tcp         0       0  127.0.0.1:8307                   127.0.0.1:61173     ESTABLISHED     34806  newreno  hostd-worker
tcp         0       0  127.0.0.1:61173                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work
tcp         0       0  127.0.0.1:80                     127.0.0.1:60974     ESTABLISHED     34267  newreno  rhttpproxy-work
tcp         0       0  127.0.0.1:60974                  127.0.0.1:80        ESTABLISHED     35402  newreno  sfcb-vmware_bas
tcp         0       0  10.2.22.101:80                   10.44.44.110:50351  TIME_WAIT           0
tcp         0       0  127.0.0.1:5988                   127.0.0.1:14341     FIN_WAIT_2      35127  newreno  sfcb-HTTP-Daemo
tcp         0       0  127.0.0.1:14341                  127.0.0.1:5988      CLOSE_WAIT    1473527  newreno  hostd-worker
tcp         0       0  127.0.0.1:8307                   127.0.0.1:45011     ESTABLISHED     34806  newreno  hostd-worker
tcp         0       0  127.0.0.1:45011                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work

NetCat

Netcat program (nc) is available on ESXi and it can test TCP connectivity to some IP target.
~ # nc -v 10.2.22.100 80
Connection to 10.2.22.100 80 port [tcp/http] succeeded!

TraceNet

Tracenet is very handy program available in ESXi to identify also latencies inside vmkernel IP stack.
~ # tracenet 10.2.22.51
Using interface vmk0 ...
Time         0.068 0.023 0.019 ms
Location:    ESXi-Firewall
Time         0.070 0.025 0.020 ms
Location:    VLAN_InputProcessor@#
Time         0.073 0.027 0.022 ms
Location:    vSwitch0: port 0x2000004
Time         0.089 0.030 0.024 ms
Location:    VLAN_OutputProcessor@#
Time         0.090 0.031 0.025 ms
Location:    DC01
Endpoint:       10.2.22.51
Roundtrip Time: 0.417 0.195 0.196 ms


Dropped packets

In this section are commands to verify dropped packets on different places of VMkernel Ip stack.
Command net-stats –l list all devices (Clients – nic-ports,vmk-ports, vm-ports) connected to VMware switch. You can simply identify to which vSwitch port number (PortNum) is device connected.
~ # net-stats -l
PortNum          Type SubType SwitchName       MACAddress         ClientName
33554434            4       0 vSwitch0         14:fe:b5:7d:8d:05  vmnic0
33554436            3       0 vSwitch0         14:fe:b5:7d:8d:05  vmk0
33554437            5       9 vSwitch0         00:0c:29:4a:5b:ba  DC01
33554438            5       9 vSwitch0         00:0c:29:f0:df:4c  VC01

Note: SubType is VM Hardware Version
vSwitch port numbers are important for following commands.
Command esxcli network port stats get –p shows statistics for particular vSwitch port.
~ # esxcli network port stats get -p 33554434
Packet statistics for port 33554434
   Packets received: 2346445
   Packets sent: 5853
   Bytes received: 295800113
   Bytes sent: 1225842
   Broadcast packets received: 1440669
   Broadcast packets sent: 336
   Multicast packets received: 896958
   Multicast packets sent: 120
   Unicast packets received: 8818
   Unicast packets sent: 5397
   Receive packets dropped: 0
   Transmit packets dropped: 0

You can also show filter statistics for ESXi firewall by command esxcli network port filter stats get –p 33554436
~ # esxcli network port filter stats get -p 33554436
Filter statistics for ESXi-Firewall
   Filter direction: Receive
   Packets in: 5801
   Packets out: 5660
   Packets dropped: 141
   Packets filtered: 150
   Packets faulted: 0
   Packets queued: 0
   Packets injected: 0
   Packet errors: 0

Filter statistics for ESXi-Firewall
   Filter direction: Transmit
   Packets in: 4893
   Packets out: 4887
   Packets dropped: 6
   Packets filtered: 6
   Packets faulted: 0
   Packets queued: 0
   Packets injected: 0
   Packet errors: 0

To show physical NIC statistics you have to use command esxcli network nic stats get –n vmnic0
~ # esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
   Packets received: 2350559
   Packets sent: 8083
   Bytes received: 312690659
   Bytes sent: 5791889
   Receive packets dropped: 0
   Transmit packets dropped: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0

Packet Capture

If you want to do deeper network troubleshooting you can do packet capturing on ESXi host. You have two tools available for packet capturing

  •  tcpdump-uw (example: tcpdump-uw –I vmk0 –s0 –C100M –W 10 –w /var/tmp/test.pcap)
  • pktcap-uw
pktcap Examples:
  • pktcap-uw –uplink vmnicX –capture UplinkRcv
  • pktcap-uw –uplink vmnicX –capture UplinkSnd
  •    you can filter for icmp –proto 0x01 or beacon probes –ethtype 0x8922

2 comments:

Jack said...

Hi David,

I work for Dell Software Group and I am trying to figure out why many of our guest VM shows large percent "receive packets dropped" number in performance chart, wondering if you can shed some light?

For example, when I check one VM's port status from the ESXi host:

esxcli network port stats get -p 33554476 | grep -i recei
Packets received: 810
Bytes received: 69103
Broadcast packets received: 767
Multicast packets received: 6
Unicast packets received: 37
Receive packets dropped: 768

But I don't see any real network issue ( this VM doesn't have much load and I have no problem to ping/ssh into it, ifconfig inside guest OS shows 0 dropped RX packets ).

The closest articles I can find are
https://communities.vmware.com/thread/469459?start=15&tstart=0
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052917

But I see same problem on our ESXi 5.5 and ESXi 6.0 host, I tried to change a VM's NIC from E1000 to VMXNET3 but didn't make any difference.

I am not sure how vmware calculate those packet numbers, does "Packets received" include "Broadcast packets received" ?

Thanks for your help,

Jack Chen


David Pasek said...

Hi Jack.

Based on your findings it seems to me like dropped packets are ARP broadcasts. The question is why it is reported in ESXi dropped packets, right? It would need deeper investigation. Do you have VMware support contract? Did you open support case with VMware? I'm very interested in this case. If you want contact me by e-mail or twitter and we can arrange call for next troubleshooting steps.