The situation has changed; the datastores are now visible but inaccessible. Each time a rescan or unmount is performed on the datastore, the ESXi hosts becomes unresponsive.
So, the issue has become critical that I have VMware support engineers trying to resolve the issue.
I believe they had issued that command but it resulted in the host being unresponsive and going into an inactive/disconnected state.
Basically, the following messages (as I mentioned before) occur over and over again:
2014-06-17T00:03:26.098Z cpu20:8212)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x28 (0x41244133e4c0, 24955) to dev "naa.600144f068d0cf00000051d578850001" on path "vmhba3:C0:T1:L41" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-06-17T00:03:26.098Z cpu20:8212)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600144f068d0cf00000051d578850001" state in doubt; requested fast path state update...
2014-06-17T00:03:26.098Z cpu20:8212)ScsiDeviceIO: 2331: Cmd(0x41244133e4c0) 0x28, CmdSN 0x29 from world 24955 to dev "naa.600144f068d0cf00000051d578850001" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
This can last for hours but will eventually timeout. From my understanding, the hostd goes under heavy load with the retries and the ESXi host cannot be connected to.
What is more concerning, although all the hosts can see the 2TB datastore(s), it sees the capacity as 0TB or 7000+TB. Also the blocksize is reported as -17609685. It looks like the datastore could be corrupt since I cannot browse it via the vSphere client. Strangely enough, the VM (Win2008R2) that has a vmdk in that datastore CAN see its data. This doesn't make any sense.
The storage is NexentaStor and I believe that someone else has run into the same issue. I have contacted Nexenta support as well and hopefully they can give some insight. Since I cannot unmount/detach the datastore, I may be forced to unpresent the LUN on the storage side.
Anyways, I'll update on our progress. Hopefully once we figure it out, this will help someone in the future.