Hello everyone! I'm in need of some assistance and would greatly appreciate and clues/ideas that could make our performance better. Here are the basic details.
We have two VM clusters Production /Non Production.
Both clusters attach to a EMC VMAX storage system with 32SSD/300FiberChannel/160Sata/512GB cache and we utilize FAST.
We have 400 LUNS (200 per environment).
Queue depth is now at 16 (EMC said we should have it at this setting)
We have about 5 VMDK's per LUN. A lot of these VMDK's are IIS/APS so they are not hit hard.
Read performance is not an issue at all. Its incredible to say the least. Write...ugg...terrible!
Performance on any physical attached to our VMAX is great.
Using IOZone on a Physical Linux I write a 4GB file with 8k block. (RH5.x) (on its own LUN, not shared)
EXT3 file system - Seq write 971MB/s
EXT3 file system - Random write 1.8GB/s
Using IOZone on a virtual (RH5.x) (On its own LUN, no other VMDK's)
EXT3 file system - Seq write 256MB/s
EXT3 file system - Random write 31MB/s
The smaller the block the worse it gets and the bigger it improves but nothing amazing. 4K blocks can yield me 10-20MB a sec. I can give you many examples if how terrible it is. Just ask and I shall capture. I have windows captures also but they are just as bad.
When I look at esxtop the HBA's do have KAVG for random in the 5 or so range and for sequential I see it grow to 40-50. There is no queueing (most likely why EMC recommend we drop it to 16).
Please let me know what I can provide to help you get an idea what may be wrong? maybe a different IO tester? different block size?
Thanks for looking!
Andy