So we just put in a new Compellent SC4020 SAN and i'm noticing that occasionally ESXi (5.5) is reporting crazy latency like anywhere from 300 to 1000+ms under performance/datastore but on the SAN side i'm seeing an average of 1-2ms (max 10) latency for the same LUN when doing large file copies or lots of small copies. I'm running 10gb to the SAN with jumbo frames, vmkping shows good connectivity with the jumbo frames and i'm not seeing any CPU spikes or anything on either the SAN or host side (when I had latency issues with my old NetApp is was usually because of CPU). Running esxtop is showing the same numbers i'm seeing via the GUI, but I guess I shouldn't be surprised there. I don't see any crazy high numbers for retransmits on the switch (Dell MXL 10/40) and no CPU or other obvious issues jumping out there. The experience within the VMs themselves seems to be just fine, so i'm not sure if there is an actual problem. Is it possible that ESX is just not reporting the numbers properly or is there something else I need to look at?

Hi All,

Using the NAS device TVS-471 :: QNAP can anyone here please share some thoughts and comments about which VMFS data store technology NFS or iSCSI would be performing better for hosting high-performance VM like Exchange, SQL and SharePoint server ?

Thanks in advance.

Here is a good comparison guide: Storage Protocol Comparison - A vSphere Perspective - VMware vSphere Blog

Hello Experts,

Actually one of my vm disk drive is giving me very bad performance ( even while copy some data from one drive to this drive) . The server is used for SAP and the drive is the db drive (oracle db). I already did some of the best practices like converted the drive to eager zeroed. But no luck. Checked with my storage team and they did not found any problem within the storage system. Got is cheked from the Os side and found no problem.  Ran the chkdsk with no errors. Any advice what next I can do to troubleshoot the problem .

Many thanks in advance.

I am building a lab for my VMware study. I have a server with two partitions. I installed VMware Player 12 and vSphere Client. I installed VMware ESXi 6 with 40GB datastore as default.

Now I need to create virtual servers on this EXSi host and I don't have enough space. I need to add new storage but neither option disk/LUN nor Network File System I could do.

How do I add storage using my existing partition on my server? either using a local disk or create a shared folder and mapping to it or any best practice you can share.

Thanks,

Using the VMware Workstation, edit your ESXi 6 virtual machine and add a new hard disk with the capacity that you want. After add the new disk, you can create a new datastore using that disk, just go to Configuration -> Storage -> Add Storage.

Hi,

Are you using jumbo frames by chance? if you are, can you ensure its enabled throughout  (vSwitch, iscsi VMkernels, physical switch, SAN, etc). If you are not using Jumbo frames, please ensure everything is set to 1500 MTU.

This error can also happen if what you are trying to present to ESXi isn't wiped already.Try creating a brand new LUN and presenting that to ESXi.

Greetings, everyone.

I encountered strange activity in my company's storage area network (BTW, I'm totally noob in this field).

In short:

ESXi hosts and VMs disk throughput rate is from 24 MiBps to 130 MiBps, although we're using HP Brocade 8 Gb switches working at 8 Gb speed and HP 3PAR StoreServ 7400 which was checked by HP tech support team for possible miconnfigurations and performance bottlenecks (none of them were found).

We have the following setup:

I made tests for the following situation:

2 servers per rack: one is ESXi 5.5.0 and other is Windows Server 2012 R2

2 LUNs per every server: one is on 7400 and other is on 7400 #2.

My results are:

Therefore I have 25-35 MiBps on LUNs which are plugged as VMFS datatores to ESXi hosts and 250-500 MiBps when LUNs are plugged to Windows server (I installed W2K12 R2 just on Blade server).

Also I tested LUNs with Crystal Disk Mark and even got 1600 MiBps.

Do you have any ideas where to start from?

Hey,

General question as I am in the process of rolling out a storage solution that will support a lightweight ESXi environment.

If I have 8 physical drives (all identical capacity and speed), would I get better performance out of 1 large array running the datastore for all or splitting the drives into 2 arrays of 4 disks each and load balancing across 2 datastores?

Thoughts?

My initial thought is that with none of the VMs being extremely I/O intensive the large disk single disc array would be better.

Are you talking about a single ESXi host with local disks ?
Then the answer is easy:
NEVER even consider using a standalone ESXi with just one VMFS-volume.
It may look like a good idea to add all disks into one large array configured with Raid5 - but resist the temptation.
When that datastore needs maintenance one day - after the next thunderstorm or the next Raid-rebuild after a diskfailure - you will regret it.
So configure a standalone host so that you have at least 2 datastores and that none of them is so large that you could not evacuate all important VMs over night.
My suggestion - not optimized for performance but for simple survival is to use datastores upto lets say 2TB and to avoid Raid5 at all costs.

If your Workstation host is able to act as NFS-server it is quite convenient to use an NFS-share.
Running VMs from that share is much more space-efficient than running the nested VMs from vmdks that are VMFS-formatted.

ESXi often has problems with creating a partition table on brand new created iSCSI-volumes.
It often helps to use a Linux to create a partitiontable first.
When ESXi sees a partitioned volume it probably also is able to repartition it.

Evening,

In Veeam, you can backup directly from a VMFS volume so long as you allow the Veeam Backup Server direct access to either iSCSI or FC LUN, it doesn't need to be mounted by the windows server, just available.

I have an environment with a Cisco UCS Chassis, where by the Fibre Channel Ports of the SAN are directly plugged into the Fabric interconnect (FC switching mode configured). The ESXi hosts are able to see the LUNS and use them as VMFS volumes.

However my veeam backup server is a virtual machine, and I would like that to be able to see the FC LUNs presented to the ESXi hosts so that I can do a direct SAN backup.

With iSCSI this is easy enough as I can add a Virtual Port Group into the iSCSI vSwitch, and add a networking adapter to the virtual machine, configuring the SAN to allow access.

With FC it looks harder, I have figured out that NPIV is enabled by the FI's by default in FC switching mode, meaning I can present an unused LUN to the ESXi hosts and the VM, and map it as a RDM. However this is not what I am trying to achieve, I want the VM to be able to see the existing LUNs used by the ESXi hosts.

Thanks in advance

We have a Synolgy SAN. It was connected to an ESXi host. After a power outage things got back into operation but the LUN did not get connected to the ESXi. We did try several things:

1. Rescan the software HBA to discover the datastore.

2. Reboot the ESXI host and then rescan the software HBA.

3. After the reboot the LUNs get detected and gets attached to the ESXI but when we try to browse the LUN through the vsphere client we do not see any contents inside the store. Although the volume of the LUN has data inside it as we can see through the Synology.

How do we get the see the contents inside the LUN?

Any help will be appreciated

Please clarify stpe 3. Do you only see the LUN under "Storage Adapters", or do you see the datastore under "Storage", but it doesn't show data anymore?

In case you can see the LUN, but the datastore isn't mounted, follow the "Command Line" steps from https://kb.vmware.com/kb/1011387 (i.e. run esxcli storage vmfs snapshot list) to see whether the LUN has been detected as a Snapshot LUN, and can be mounted manually.

André

Hi.

Does anybody has experiences if it is possible to use a hardware vss provider

for a san storage from inside a VM for a physical RDM?

What we want to do is to create a san based snapshot of a RDM LUN

from inside a VM and after that present that snapshot to a different

server.

Any suggestions are most appreciated.

Thank

goppi.

Buenos días,

Tengo un problema con uno de mis discos virtuales SERVER-AD_1-000001.vmdk de 1.4TB de tamaño, dejo de reconocer el Vmware Esxi 6.0, los sistemas lo tenia en otros discos virtuales, al intentar levantar SERVER-AD_1-000001.vmdk me salia un error de CID mismatch error, que al parecer era por una incongruencia en los CID y averiguando aplique una linea de comando.

dd if=/vmfs/volunes/5535cd-xxxx/SERVER-AD/SERVER-AD_1-00001.vmdk count=10 | sed 's/parentCID=fbfacd4f/parentCID=bc72a0e2/'>SERVER-AD_1-000001.vmdk

Despues de aplicar este comando el disco virtual SERVER-AD_1-00001.vmdk de 1.4Tb paso pesar 0kb, y despues de esto aparecio un archivo SERVER-AD_1-000001Delta.Vmdk de 1.4Tb, quisiera saber si podre recuperar la información que estaba en ese disco.

El SERVER-AD_1.vmdk al parecer es un snapshop que hice hace 1 año pero solo esta la data de ese año que pesa solo 300gb.

Como puedo recuperar  la informacion que estaba en SERVER-AD_1-000001.vmdk  (Solo contenia archivos y carpetas importantes)

Gracias por su ayuda.

Atentamente

Victor Sarmiento

HI a.p,

By Step-3, I mean I had restarted the ESXi host and tried to reconnect the LUN (with its iqn number given in the SAN). The LUN gets connected to the esxi host but this LUN did have my VM's and as I use the vsphere client to browse through the LUN it shows me moting. It get detected as a new LUN without display the contents.

Following is the log that I get from /var/log/vmkernel.log

####

2016-08-14T03:24:31.844Z cpu3:33597)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:25:01.845Z cpu5:38173)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:25:31.845Z cpu5:38173)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:01.846Z cpu5:38172)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:31.847Z cpu4:33597)WARNING: NFS: 221: Got error 13 from mount call

2016-08-14T03:26:38.174Z cpu2:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.249Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.356Z cpu0:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.356Z cpu0:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.356Z cpu0:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.357Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.366Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.368Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.370Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.372Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.374Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.378Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1958: Device rescan time 92 msec (total number of devices 10)

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1961: Filesystem probe time 143 msec (devices probed 7 of 10)

2016-08-14T03:26:38.424Z cpu1:34366)VC: 1963: Refresh open volume time 13 msec

2016-08-14T03:26:38.429Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.462Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.482Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.487Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.489Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.491Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.493Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.495Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.499Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.499Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.499Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.499Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1958: Device rescan time 44 msec (total number of devices 10)

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1961: Filesystem probe time 107 msec (devices probed 7 of 10)

2016-08-14T03:26:38.584Z cpu7:34366)VC: 1963: Refresh open volume time 2 msec

2016-08-14T03:26:38.654Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.686Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.753Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.753Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:38.753Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:38.753Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.759Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.761Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.763Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.765Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.767Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:38.771Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1958: Device rescan time 79 msec (total number of devices 10)

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1961: Filesystem probe time 112 msec (devices probed 7 of 10)

2016-08-14T03:26:38.848Z cpu4:34366)VC: 1963: Refresh open volume time 2 msec

2016-08-14T03:26:41.731Z cpu1:35082)LVM: 13082: One or more LVM devices have been discovered.

2016-08-14T03:26:55.336Z cpu0:34366)LVM: 7525: Initialized naa.600140574ab763dde2dfd493adb910de:1, devID 57afe4ff-89c5a615-b2da-0026b94357b6

2016-08-14T03:26:55.343Z cpu4:34366)LVM: 7613: Zero volumeSize specified: using available space (2060491668992).

2016-08-14T03:26:56.717Z cpu5:34366)FS3: 224: <START fbb>

2016-08-14T03:26:56.717Z cpu5:34366)1964800 resources, each of size 0

2016-08-14T03:26:56.717Z cpu5:34366)Organized as 154 CGs, 64 C/CG and 200 R/C

2016-08-14T03:26:56.717Z cpu5:34366)CGsize 65536. 0th CG at 65536

2016-08-14T03:26:56.717Z cpu5:34366)FS3: 226: <END fbb>

2016-08-14T03:26:56.939Z cpu3:34366)FS3: 224: <START fdc>

2016-08-14T03:26:56.939Z cpu3:34366)130000 resources, each of size 2048

2016-08-14T03:26:56.939Z cpu3:34366)Organized as 11 CGs, 64 C/CG and 200 R/C

2016-08-14T03:26:56.939Z cpu3:34366)CGsize 26279936. 0th CG at 65536

2016-08-14T03:26:56.939Z cpu3:34366)FS3: 226: <END fdc>

2016-08-14T03:26:56.980Z cpu1:34366)FS3: 224: <START pbc>

2016-08-14T03:26:56.980Z cpu1:34366)64512 resources, each of size 4096

2016-08-14T03:26:56.980Z cpu1:34366)Organized as 63 CGs, 64 C/CG and 16 R/C

2016-08-14T03:26:56.980Z cpu1:34366)CGsize 4259840. 0th CG at 65536

2016-08-14T03:26:56.980Z cpu1:34366)FS3: 226: <END pbc>

2016-08-14T03:26:57.088Z cpu0:34366)FS3: 224: <START sbc>

2016-08-14T03:26:57.088Z cpu0:34366)32000 resources, each of size 8192

2016-08-14T03:26:57.088Z cpu0:34366)Organized as 8 CGs, 64 C/CG and 64 R/C

2016-08-14T03:26:57.088Z cpu0:34366)CGsize 33619968. 0th CG at 65536

2016-08-14T03:26:57.088Z cpu0:34366)FS3: 226: <END sbc>

2016-08-14T03:26:57.144Z cpu3:34366)FS3: 224: <START pb2>

2016-08-14T03:26:57.144Z cpu3:34366)256 resources, each of size 4096

2016-08-14T03:26:57.144Z cpu3:34366)Organized as 1 CGs, 64 C/CG and 16 R/C

2016-08-14T03:26:57.144Z cpu3:34366)CGsize 4259840. 0th CG at 65536

2016-08-14T03:26:57.144Z cpu3:34366)FS3: 226: <END pb2>

2016-08-14T03:26:57.173Z cpu0:34366)Res3: 5734: SDDir: type: 0x5, fileLength: 0x118, numBlocks: 1

2016-08-14T03:26:57.176Z cpu0:34366)Vol3: 3434: Created VMFS-5.60 with config 0x16 on vol 'iSCSI-Mel-Black-SAN-1-Ergo-VM'

2016-08-14T03:26:57.176Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.228Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.381Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.386Z cpu0:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.388Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.390Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.392Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.394Z cpu1:34366)<3>ata3.00: bad CDB len=16, scsi_op=0x9e, max=12

2016-08-14T03:26:57.398Z cpu1:34366)FSS: 5051: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0': Not supported

2016-08-14T03:26:57.398Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:57.398Z cpu1:34366)Vol3: 714: Couldn't read volume header from control: Not supported

2016-08-14T03:26:57.398Z cpu1:34366)FSS: 5051: No FS driver claimed device 'control': Not supported

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1958: Device rescan time 68 msec (total number of devices 10)

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1961: Filesystem probe time 153 msec (devices probed 7 of 10)

2016-08-14T03:26:57.415Z cpu1:34366)VC: 1963: Refresh open volume time 15 msec

2016-08-14T03:26:57.514Z cpu1:34366)Vol3: 661: Unable to register file system iSCSI-Mel-Black-SAN-1-Ergo-VM for APD timeout notifications: Already exists

2016-08-14T03:26:57.514Z cpu1:34366)LVM: 13062: File system '[iSCSI-Mel-Black-SAN-1-Ergo-VM, 57afe500-482444ee-1305-0026b94357b6]' (LV 57afe4ff-5820eaf4-7436-0026b94357b6) mounted in 'rw' mode.

2016-08-14T03:26:58.341Z cpu0:38173)LVM: 13082: One or more LVM devices have been discovered.

####

Just to mention. I am running VMware ESXi 5.5.0 and there are no snapshots in any of the VM's.

I just want to reconnect the LUN to the host and see its contents. SAN volume shows that it has data inside it.

The naa-id is naa.600140574ab763dde2dfd493adb910de:1

I ran the command:

esxcli storage vmfs snapshot list

and it does not return anything.

That command will very likely produce an empty file as you should not have SERVER-AD_1-00001.vmdk at all.
Count the zeros in the filename !

It happens occasionally that a LUN looks empty after a power failure.
Be careful with your next steps.Probably your VMs are still there and can be extracted if you dont make mistakes.
Please create a VMFS-header dump with
dd if="/dev/disks/naa.600140574ab763dde2dfd493adb910de:1" bs=1M count=1536 of=/vmfs/volumes/some-other-datastore/day3_noc.1536
Download the *1536 file, compress it and provide a download.
With that info we can tell you how serious it is.
Ulli