Hello folks,
Subject says it all. Here's the scoop:
Back end: IBM Storwize V7000, two canisters, each one with 2 x 10Gb connections (total of 4 x 10Gb connections)
Front end: ESXi 5.5 U1., configured with iSCSI,
2 x 10Gb Intel 82599 network cards;
2 vmdk portgroups for iSCSI, each bound to a different physical adapter as per best practices;
MTU is enabled at 9000 all the way to the storage.
I initially configured the host with FC to the same storage. Using a single VM running IOmeter, I was able to get 64,000 IOPS from SAS drives and 96,000 IOPS from SSD drives. Thus, I know my back end can provide 64,000 / 96,000 IOPS.
Upon replacing the HBA with the iSCSI software adapter, I get a mere 46,000 IOPS for BOTH SAS and SSD. I was expecting a drop, but not that much. Also, the fact that SAS and SSD run at the same speed gives me the pointer that it's not the storage, it is the medium.
Here's what we've tried:
- it is not the path: we suppressed the network switches (Nexus 5K) and plugged the storage directly into the ESXi server.
- We played disabling/enabling Delayed ACKs.
- We played disabling/enabling TSO/LRO
- We tuned the queue depth
- We tuned the ixgbe driver for the Intel 82599, disabling interrupt moderation
- We played with multipath: disabled it, re-enabled it, changes the IOPS rotation to 1.
- We changed the TCP/IP stack heap size to give it more room.
- Upon monitoring, we don't see the ESXi network interfaces as being busy, nor the iSCSI heads on the other end being busy. Low network, low CPU, and low Storage utilization on the SAN.
The only think I can come up with is the TCP/IP stack on ESXi. The stack is optimized for VMs data transfer, and I think vmdk traffic goes to a second place. Also, I haven't found any optimization for TCP/IP Window size and Segment size (as Windows has).
Is there anything that you folks can come up with? Any insight will be really appreciated.
Thanks all,
Pablo