It is not uncommon to see network performance degradation on a ESX guest when a high number of connections occur, or the workload is increased. One might observe high network latency, network packet loss, high CPU usage and/or application time-out issues. From SQL Server, the issue can be detected by observing a high number of ASYNC_NETWORK_IO waittype. Although there are no problems with the SQL Server configuration, a developer or the application engineering team indicates a network related issue, others may conclude that the root cause is bad application design and coding practice.
While the application code may have some fault, it could also be the VMXNET3 drive configuration on the VMWare guest that needs to be tweaked.
Performance Issues from VMXNET3:
The VMWare VMXNET3 is an enhanced and feature rich network driver, however it can be problematic if the driver is not optimally configured. Following are a few performance issues worth noting:
- Network packet loss
- High network latency (slow data transfer)
- High CPU usage on the guest
- Disconnection of client applications
- Unexpected cluster failover when nodes are built on Virtual Machines
SQL Server Symptoms:
Following are some symptoms from SQL Server’s side:
- Long running queries
- Excessive ASYNC_NETWORK_IO waittype
- Query time-out
- High CPU usage or CPU Spike
Recommendations for VMXNET3:
It is highly recommended to adjust the VMXNET3 network driver configuration on the guest system to improve network performance. The configuration changes on VMXNET3 is for all Windows based guest Operating Systems, regardless of whether a guest system experiences issues or not. Following are a few recommended changes that need to be considered and adjusted:
- Small Rx Buffers: Increase the value to its maximum, which is 8192.
- Rx Ring #1 Size: Increase the value to its maximum, which is 4096.
- Receive-side scaling (RSS): Enable RSS on the VMXNET network adapter.
- Chimney Offload state: Disable this option if it is not already disabled.
- Power Plan: Make sure that the High performance option is selected in the power plan (run powercfg.cpl).
- Speed & Duplex: Make sure that Auto-Negotiation of the VMXNET3 is detecting the network bandwidth properly.
Receive Side Scaling (RSS):
This network driver configuration within Windows Server enables distribution of the kernel-mode network processing load across multiple CPUs. If it is disabled, then all network processing will be handled by a single CPU core and could lead to network bottleneck.
By default, Windows uses up to 4 CPUs for RSS and the maximum RSS value is 16. However, the value should not exceed the total number of cores recorded on a Windows Server.
Configuring a VMXNET3:
On a VMWare guest, use the following command to check the status:
################################################
# get VMXNET3 information
################################################
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Get-NetAdapterAdvancedProperty |`
Format-Table -AutoSize Output from the above script:
Querying TCP status:
To reconfigure the VMXNET3, we can use the following PowerShell Script for the targeted NIC Adapter. Please note that this will cause a NIC disconnection during the Restart-NetAdapter call, and will interrupt the network connection on the adapter for several seconds.
PowerShell Script:
################################################
# Enable RSS globaly and disable Chimney Offload
################################################
Set-NetOffloadGlobalSetting -ReceiveSideScaling Enabled
Set-NetOffloadGlobalSetting -Chimney Disabled
################################################
# get VMXNET3 information
################################################
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Get-NetAdapterAdvancedProperty |`
Format-Table -AutoSize
################################################
# Reconfigure VMXNET3 settings
################################################
# Receive Side Scaling
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Receive Side Scaling' -DisplayValue 'Enabled' -NoRestart
# Receive Throttle
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Receive Throttle' -DisplayValue '30' -NoRestart
# Small Rx Buffers
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Small Rx Buffers' -DisplayValue '8192' -NoRestart
# Rx Ring #1 Size
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Rx Ring #1 Size' -DisplayValue '4096' -NoRestart
<#Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -RegistryKeyword 'MaxRxRing1Length' -DisplayValue '4096' -NoRestart #>
# Rx Ring #2 Size
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Rx Ring #2 Size' -DisplayValue '4096' -NoRestart
<#Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -RegistryKeyword 'MaxRxRing2Length' -DisplayValue '4096' -NoRestart #>
# Large Rx Buffers
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Large Rx Buffers' -DisplayValue '8192' -NoRestart
# Speed & Duplex
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Set-NetAdapterAdvancedProperty -DisplayName 'Speed & Duplex' -DisplayValue '10 Gbps Full Duplex' -NoRestart
# Restart the VMXNET3
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Restart-NetAdapter
################################################
# To reset VMXNET3 settings to default
################################################
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like 'VMXNET3*' } |`
Reset-NetAdapterAdvancedProperty -DisplayName '*' Reconfiguration of VMXNET3:
VMXNET3 Properties:
Windows Power Plan:
Conclusion: Applying best practices around VMXNET3 will definitely help to improve overall performance, but any performance issues related to the application will not be completely eliminated. The optimization efforts towards the application code will dramatically reduce network related performance problems, and hence the reduction of the waittype ASYNC_NETWORK_IO.
References and further reading:
- ASYNC_NETWORK_IO: https://www.sqlskills.com/help/waits/async_network_io/
- Large packet loss at the guest operating system level on the VMXNET3 vNIC in ESXi (2039495): https://kb.vmware.com/s/article/2039495
- Poor network performance or high network latency on Windows virtual machines (2008925): https://kb.vmware.com/s/article/2008925
- Setting the Number of RSS Processors: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/setting-the-number-of-rss-processors
- TCP Chimney Offloads and SQL Server Implementation: https://sqltouch.blogspot.ca/2013/03/tcp-chimney-offloads-and-sql-server.html
