As part of the re-build I had decided to implement a couple of virtual machine clusters, one for a scale-out file server and one as a SQL cluster.
I'd deployed the virtual machines for the cluster nodes using Service Templates in SCVMM and as part of that template chosen to use an availability set to ensure the VM's were separated across hosts (a cluster doesn't provide much High Availability if they all reside on the same host that's failed!).
When I started to create the cluster I ran straight into a problem with the Failover Cluster Manager reporting problems due to timeouts when creating the cluster.
Creating a single node cluster worked fine, but then would again fail when trying to add in another node.
I happened to put one of the Hyper-V hosts into maintenance mode for something and it migrated all the VM's onto the same host, at which point creating the cluster worked flawlessly, yay!
However, when the Hyper-V host came back out of maintenance mode and the availability sets kicked in during optimisation forcing a VM node back onto a separate physical host, the clusters broke again, not yay :(
So after some
Sophia_whx suggested trying to use Disable-NetAdapterChecksumOffload on the NICs to help with the issue.
So, first things first. Use the Get-NetAdapterChecksumOffload to see just what the configuration was and sure enough Checksum Offload is enabled for just about all services across the majority of the NICs.
I then used the Disable-NetAdapterChecksumOffload * -TcpIPv4 command which resulted in this:
After doing this on the other host and giving them a reboot, my clustered virtual machines appear to be nice and stable when split across physical hosts. Yay! Problem fixed.
Just as another side note about Broadcom adapters, there have also been reports of performance issues when the Virtual Machine Queue (VMQ) setting is enabled, despite it being a recommended setting.
A quick check of my hosts showed it was enabled:
Another quick PowerShell line later and it wasn't:
Get-NetAdapterVmq -InterfaceDescription Broad* | Disable-NetAdapterVmq