High IO/Latency on HCI S2D Cluster

[Guest /u/cptkommin]

Good Day

 

We are experiencing an issue with our HCI S2D Cluster where our storage/CSV performance degrades rapidly once a node is taken offline, ie for Windows Patching. We see massive (Seconds latency) on our CSVs, causing the hosted VMs to start to fail, and causing data corruption. We also see random high write latencies during the day. In Event Viewer, the only event ID we find that claims anything to high latency is an EventID 9 under Hyper-V-StorageVSP Channel:

 

"An I/O request for device 'C:\ClusterStorage\3WM_CSV03\Virtual Machine Name\Virtual Hard Disks\Virtual Machine Name - C_Drive.vhdx' took 24040 milliseconds to complete. Operation code = SYNCHRONIZE CACHE, Data transfer length = 0, Status = SRB_STATUS_SUCCESS."

 

Our Setup:

5x Dell R740xd

 

Each node has the following:

 

1.5TB DDR4 3200 RAM

 

2 x 6.4TB MU Dell Enterprise NVMe (Samsung)

 

10x 8TB SAS 12Gps 7.2k Dell Enterprise spindle disks

 

2x Intel Gold 5220R CPUs

 

2x Intel 25G 2P E810-XXV NICs

 

All 5 nodes are set up in an S2D cluster. The NVMe serves as the cache and the spindles as the storage. The cluster is set with an in-memory cache value of 24GB per server. Storage repair speed is set to high, dropping this to the recommended medium speed does not make any difference. Cache mode is set to Read/Write for both SSD and HDD in the config. The cache page size is 16kb and the cache metadata reserve is 32GB. On a Hyper-V level, we have enabled NUMA Spanning. We have five 3-way mirror CSVs in the storage pool. Networking consists of a SET Switch, 5 virtual networks (1x management, 1x backups, 4xRMDA ). We have 2x Dell S5248F switches servicing the core/physical network. Adaptors are set up with Jumbo packets enabled, VMQ and VRSS, iWARP, and no SR-IOV.

 

Firmware/Drivers are mostly up to date, but this has not proven to be of any help. In fact, we are running v22.0.0 (Dell versioning) firmware/drivers for the NICs as it has proven to be stable, ie not causing the host OS to BSOD.

 

We were running Server 2019 when we first encountered this issue. After months of back and forth with MS Premier Support, the solution was to upgrade to Server 2022 due to the improvements in S2D/ReFS. We complied and started the upgrade process. Initially, two nodes were removed and reloaded with WS22, and everything was configured as stated above, with one exception: the CSVs were a 2-way mirror since only 2 nodes were present in the cluster. We started migrating VMs, added the 3rd node, and created the first 3-way mirror CSV; all is still well and dandy. We continued with this until we had a full 5 node '22 HCI S2D Cluster, and then give or take 3-4 months in, we started experiencing the exact same issue. I must add, not as high latencies as in WS19, but still high enough to cause a VM to crash and corrupt data.

 

We have another MS Premier Support ticket open, and as you can imagine, they have no clue what the issue could be. We have uploaded probably close to 1TB worth of TSS logs/Cluster Event logs etc, and still no step closer to a cause or some sort of solution. Dell Support is of no help since none of the Dell Support TSR logs show anything. No failed hardware, no warnings, or errors, ie a failed drive, etc.

 

This effectively prohibits us from doing any "live" maintenance as anything could potentially cause high IO/latency, and when we want to schedule maintenance for patching, we need to shut down all clustered services, which is a nightmare to try and schedule with clients every month.

 

We have a suspicion that it could be network related, as in the RDMA/NIC config might not be optimal, and the sync/repair job on the storage pool starts queuing, causing the increase in IO/latency across the board. Happy to share our PS config via DM with any experienced engineers out there. We have the Intel PROSet software installed and are considering redoing the SET Switch, and enabling SR-IOV, and making use of the Virtualization Profile in the PROSet software, but we are still researching this avenue.

 

I would appreciate any suggestions that can help in finding the cause and resolving this case. If there is any other/more info I can share, let me know.

 

Much appreciated.

 

 

submitted by /u/cptkommin

[link] [comments]

 

Continue reading...