The hosts are on U1 Build 702118 and yes it is Gen 7 or 8 .. I will have to check that as it has been removed for now - but I am intrigued - what would the fix be because at the moment we are reluctant to put it back into production.
As for a solution - a week now down the line and we are still in the middle of a blame war, but I THINK I have fixed the issue for now. Oh and a storage processor reboot is planned for next week already. We still have the odd issue, with standby luns missing, some hosts missing some LUNs etc., which is hopefully solved by the reboots.
But the high latency was the main concern of course, but that I think is fixed at the moment.
How ? Well it doesn't look like it was storage after all (regardless of VMware insisting on it up until now).
Like I said - the hosts behave fine when in maintenance mode and only one LUN was affected.
Host in cluster = High latency on LUN X
Host rebuilt and put back in cluster = High latency on LUN X
Put host in maintenance mode = Latency gone.
Take host out of maintenance mode = High latency on LUN X
Remove host from cluster = Latency gone
Remove host from vCenter = Latency gone
So it smelt fishy really and if it was the storage, then why is the issue gone in the conditions above? I am still waiting for the answers, anyway, what is involved in the conditions where the latency is high ? The vCenter / HA agent.
Checked HA and bang - the LUN in question was used for HA heartbeating so yes, your guess was also spot on (a shame I only read it today, could have saved me a night of tinkering).
I then remembered that HA didn't work when the host crashed / powercycled so a theory is, the heartbeat folder was locked even though it wasn't ... or wasn't even though it was supposed to.
Either way - when change the HA datastore heartbeat to another LUN - the latency issue disappeared ...
So it clearly was a VMware issue after all, and not an issue on the storage itself. It really is frustrating that VMware support still has no idea and still blames the storage vendor and vice versa ...
The only option is obviously Business Critical Support with onsite support - but we cannot really afford the $50k / year (or whatever that costs nowadays) - so we are at the mercy of VMware and other vendors involved.
Either way - I'd appreciate any further hints regarding the Gen 7 / 8 blades