Hi Community
First, let me give you a brief overview of our small infrastructure:
1 x HP Enclosure C7000
6 x HP BL460 G7/Gen8 blades with ESXi 5.1
1 x HP BL 460 G7 with Windows 2008 R2
2 x HP B-Series 8/24 switches
1 x HP P2000 G3 FC Dual Controllers
Here is the story - about 4 weeks ago one of the controllers in our MSA failed. During the replacement of the controller we had issues when both controllers believed their were owners of vdisks.
Once the controller was replaced I noticed very poor performance during the storage vmotion to one particular datastore.
On the MSA we have two vdisks with 10K drives:
vdisk-1 - 16 x 600GB 10K disks in RAID6
vdisk-2 - 8 x 600GB 10K disks in RAID6
Both vdisks are owned by the new controller and each contains one datastore.
When i move VM from vdisk 1 to vdisk 2 it works as expected.I can see about 150-200 MB/s rate.
However, when I move the VM in the opposite direction, from vdisk 2 to vdisk 1 it takes 5 times longer and the latency goes really high, up to 1000ms and throughput is about 30MB/s.
In my opinion there are some issues with Write IOs on vdisk-1.
The problem started right after the controller was replaced. Therefore, I first blamed HP. I have also double checked everything I could remember - SAN cabling and zoning, ALUA, VAAI plugins, but results are the same. MSA log has no errors/warnings.
However, when I presented two new volumes, one from each vdisk, to the blade server running Windows and I have had no performance problems there copying files both directions.
I have also shutdown the new controller to fail over all vdisks to the second controller, but that didn't resolve the performance problem.
HP Support couldn't find anything wrong with the storage. With all that in mind it is very probably that the storage is just fine and there are some issues with vSphere.
We have another couple of vdisks on the same MSA owned by the second controller and with numerous volumes presented to ESXi hosts. Storage VMotion works fine on them.
To be honest I ran out of ideas and I will really apreciate any tips on where to look for the cause of the problem.
Thanks.