(Part 2)

Troubleshooting Hyper-V Network Virtualization 

Introduction

The first part of this article series provided an overview of two troubleshooting approaches you can follow to troubleshoot issues with Hyper-V network virtualization. The first troubleshooting approach helps identify issues with the Hyper-V network virtualization configuration on Hyper-V hosts, and the second troubleshooting approach, which is an advanced approach, helps you figure out issues by enabling tracing on the Hyper-V network virtualization components, which is explained in the later section and final part of this article series.

In Part I, we learned some of the misconfiguration possibilities, which might cause communication failures between virtual machines participating in Hyper-V network virtualization. In the second part of this article series, we continue to look at some more possibilities.

Since we have used an example of Hyper-V network virtualization design for the troubleshooting purposes in the first part, I would recommend you reading the first part of this article series to familiarize yourself with the scenario we are troubleshooting throughout the article series.

• Wrong Provider Address is assigned

A Provider Address must be assigned to all the Hyper-V Hosts. The PA Address is assigned to the NIC of the virtual network switch to allow virtual machines to communicate at the physical layer. A Provider Address can be assigned either using SCVMM or HNV PowerShell cmdlet. In case a Provider Address is not assigned, all virtual machine communication will stop. To list provider addresses configured on HVHOSTA and HVHOSTB, execute the command below on both the Hyper-V hosts.

• Get-NetVirtualizationProviderAddress | FT ProviderAddress, PrefixLength, InterfaceIndex –AutoSize

The output returned from the above command is shown below for HVHOSTA:

And when you run the above command on HVHOSTB, the output should look like this:

 

As you can see in outputs, returned from HVHOSTA and HVHOSTB, both Hyper-V hosts are assigned with the correct Provider Addresses. If the output is not returned as expected, you should check the SCVMM PA Pool to ensure correct Provider Addresses are configured for HVHOSTA and HVHOSTB Hyper-V hosts if you implemented HNV using SCVMM.

In the same output, you must also ensure that Provider Addresses are assigned to the correct NIC of the virtual network switch by looking at the InterfaceIndex number. The InterfaceIndex number indicates the NIC to which the virtual network switch is mapped.

• VM1 and VM3 are not connected to a Virtual Switch

Virtual machines do not disconnect from the Virtual network switches themselves. However, it is a good practice to ensure that the virtual machines (VM1 and VM3 in this case) are connected to a virtual network switch, which is configured for Hyper-V network virtualization.

• MAC Addresses are not unique

When virtual machines are deployed using SCVMM, SCVMM ensures that the unique MAC addresses are assigned to the virtual machines. In case you face any connectivity issues, always check to make sure unique MAC addresses are assigned to each virtual machine. You can execute Get-VMNetworkAdapter PowerShell cmdlet to obtain MAC addresses assigned to each virtual machine.

• WNV is not bound to the Virtual Adapter

WNV (Windows Network Virtualization) module must be bound to the virtual adapter on all Hyper-v hosts. To check if WNV is bound to a network adapter, execute the following command on both Hyper-V hosts:

• Get-NetAdapterBinding | Where {$_.ComponentID –eq “MS_NetWNV”} | FT Name, InterfaceDescription, ComponentID -AutoSize

By running the above command, you will see the below output for both Hyper-V hosts:

Tip:
Since WNV Module is bound automatically on Windows Server 2012 R2, there is no need to perform this step on Hyper-V running on Windows Server 2012 R2.

• Customer Route is missing

A customer route is created for each virtual subnet. These routes provide information to route traffic between virtual subnets within a VM Network. To list the customer routes, execute Get-NetVirtualizationCustomerRoute PowerShell cmdlet on both Hyper-V hosts.

• Get-NetVirtualizationCustomerRoute | FT VirtualSubnetID, DestinationPrefix, NextHop –AutoSize

The output returned from both Hyper-V hosts are shown below:

As you can see in the above output, customer routes exist for both virtual subnets (10.0.0.0/24 and 10.0.1.0/24). It can be assumed that there are no issues with the customer routes here.

• Virtual Machines are using the Reserved IP Address for the Virtual Subnet Gateway

Virtual Machines, participating in the Hyper-V network virtualization, must not use the reserved IP address for the virtual subnet gateway. In HNV, the “.1” address of a virtual subnet is reserved by HNV to route traffic between virtual subnets.

• Physical machines are using the Provider Addresses assigned to Hyper-V hosts

A Provider Address must not be used by any other physical machine running on the same network. Provider Addresses can be assigned either using SCVMM or PowerShell cmdlet. SCVMM ensures that unique Provider Addresses are assigned to Hyper-V hosts. These Provider Addresses are assigned from the PA Pool configured at the SCVMM. Make sure that no physical machines on the network are using these addresses.

• Virtual Machines are assigned with the VLAN ID

Please note that Virtual Machines participating in Hyper-V Network Virtualization must use VSID. You must check on the property page of virtual machines to ensure that VLAN IDs are not assigned. In case virtual machines are assigned with a VLAN ID, please remove it.

• Updates via VMM Server were not received by Hyper-V Servers

The VMM server is responsible for automatically updating each Hyper-V host with updated configuration of Hyper-V network virtualization. If you find an inconsistency in the HNV configuration, you must resolve the configuration issue within the SCVMM. SCVMM must have connectivity to Hyper-V hosts to ensure that the updated configuration is applied during the next polling cycle, which is every 10 minutes.

If you manually make any configuration changes on the Hyper-V hosts, the manually added settings will be overwritten by VMM during the next polling cycle.

If you don’t see any configuration issues and end up doing further research, it is the time to enable tracing on the Hyper-V network virtualization components, which is explained in the next section of this article series.

• Enabling Tracing on Network Virtualization Components

One of the new features of Windows Server 2012 is the “Unified Tracing”. Windows Server 2012 and, later Operating Systems, has a new NetSh parameter, which can be used to capture virtual machine traffic. The “Unified Tracing” can be enabled by executing the NetSh command line tool. The “Netsh trace” context includes pre-defined sets of trace providers. These providers are known as scenarios, which you can enable for troubleshooting.

To view a complete list of scenarios, as well as a brief description of each scenario’s purpose, you can type “netsh trace show scenarios” command in the command prompt. Microsoft provides “Microsoft-Windows-WNV” as a provider for tracing events related to Hyper-V network virtualization and provider “Microsoft-Windows-Hyper-V-VmSwitch” traces events related to the virtual network switch.

In Hyper-V Network Virtualization, the virtual traffic leaves the virtual machine. Hit the Network Virtualization WNV Module, and then hit the virtual network switch. Since the Hyper-V network virtualization traffic hits both the virtual network switch and the Windows network virtualization module, and only one NetSh trace can be active at a time, it is necessary to enable tracing on both the providers together.

As a high-level process, please follow the steps below to troubleshoot issues using “Unified Tracing”:

1. Enable Unified Tracing on both providers to capture events to an ETL file.
2. Reproduce the connectivity failure issue, or the issue being investigated.
3. Stop the trace.
4. Convert the ETL file to a text file.
5. Finally, analyze the output.

We will run through these steps in detail during the final part of this article series.

Summary

The second part of this article series helped you learn a few more possibilities, which might cause VM1 and VM3 to stop communicating with each other. We also provided an overview of the second troubleshooting approach, which is to enable “Unified Tracing” on the Hyper-V network virtualization components.

In the final part of this article series, we will see how enabling “Unified Tracing” on network virtualization components can help you solve communication issues between virtual machines.