I am working at a recovery case that looks really really strange.
When I was called everything looked like a normal case of the "my datastore suddenly lost all VMs" issue.
And indeed - looking at the datastore using ESXi itself - all that was visible were the 6 hidden .*.sf files and the one .sdd.sf directory.
I am used to that so I followed my usual approach and looked at the datastore with vmfs-tools (using a good build and not the outdated version thatz comes with Ubuntu)
Same result - only the sf - metadatafiles were visible.
Next I made a VMFSheader-dump and went home for a more detailed analysis.
Inside the header-dump I then found 3 vmx-files and 6 vmx-backups, 1 vmsd, 2 vmxf, 3 descriptors for basedisk, 3 earlier versions of those descriptors and one descriptor for a snapshot.
With that files I deduced that the content of the datastore must have looked like this:
a 2k3-VM with a 200gb disk + one snapshot
a 2k8 R2 VM with a 200gb vmdk
a 2k8 R2 VM with a 150gb vmdk
Next I searched the filedescriptor-section but only found the hidden sf files.
At this stage a first solid prognosis of the chances for a successful recovery can be made.
In this case the result was:
3 VMs were on the datastore: all small descriptorfiles were found
2 thick flat files missing, one thin flat file missing and one delta of unknown size missing.
Without the entries in the filedescriptorsection no easy automatic recovery is possible.
The customer then asked for the 2k3 VM with the thin basedisk and the snapshot.
I told him that I would try but that the result would be unpredictable.
After a search for the inodes I then found 4 large orphaned files with a high fragmentation rate.
And to my surprise I really could extract the 4 large files.
Even more surprising - the thin vmdk could be extracted with 120000 dd-commands - the delta needed 71000 dd-commands, the two thick vmdks needed 51000 and 7100 dd commands.
I had not expected any good results for the thin vmdk - but when I tested it - it survived the first boot and not even required a checkdisk. The snapshot was accepted as valid when I attached it to the thin vmdk. the two thick vmdks also looked good.
So until this point everything worked as expected - and actually much better than expected.
The really strange results that made me raise this question appeared when I started to check the thin flat with the snapshot.
- the basedisk alone looked absolutely healthy - a typical 2k3 server was bootable without errors - the state was from 2010 ! - so the snapshot must have been in use for 6 years.
- after I attached the snapshot the 2k3 server no longer was bootable and after checking with a LiveCD I found that the NTFS-partition was completely damaged - the MFT was completely missing.
This result was surprisingly bad - so I hexdumped the delta and found that it included a MFT.
Surprised by this result I tried the same snapshot attached to a new, fake basedisk that I created with
dd if=/dev/zero bs=1M count=1 of=fake-basedisk-flat.vmdk
dd if=/dev/zero bs=1M count=1 of=fake-basedisk-flat.vmdk seek=$((200*1024)) conv=notrunc
This fake basedisk I then quick formatted with NTFS.
Basically I tried to attach a delta that I considered as damaged to a new basedisk with an empty but new NTFS filesystem.
I expected a similar result as with the original basedisk but what I saw next completely took my by surprise.
Instead of a damaged NTFS partition a complete 2k3 system appeared and the files had timestamps ranging from 2010 upto June 2016.
A quick check of the health state of the filesystem came up as 180000 good files, 27000 good folders versus 5 bad files and 20 bad folders.
The damaged files were located in "program files\vmware\vmware tools\guest sdk\ and Windows\Pchealth\helpctr\
Another big surprise was the timestamp of the $MFT which was from 2010 !
The big questions that now come up are:
Why are the results of the 2 attempts to read one delta so very different:
total corruption when I use delta + original flat
surprisingly good when I use delta + new fake basedisk
Does it make sense to carve out a thin vmdk with 120000 dd commands that neither have any line referencing /dev/zero nor any gaps ?
- do I have to assume that the assumption to have a thin vmdk is wrong ?
- do I have to assume that my 120000 dd commands have errors ?
- can the line ddb.thinProvisioned = "1" in a descriptorfile be regarded as unreliable when checking wether the state of a vmdk is thick or thin ?
Has anybody ever tried to manually extract data from a delta.vmdk with a series of dd-commands and a list of offsets ?
Does anybody know reliable checks to check wether an unnamed delta.vmdk belongs to another unnamed flat.vmdk ?
I am not sure if I managed to explain the case good enough - if you have any theory that could explain the strange behaviour - please let me know.
Thanks
Ulli
Maybe the inode-details are useful ?
This is the delta
03809804-InodeIDdec=58759172
03809804-InodeIDhex=03809804
03809804-InodeID2= 15
03809804-InodeNLink= 1
03809804-InodeType= 3
03809804-InodeType=file
03809804-InodeFlags= 0
03809804-InodeSize=3490072576
03809804-InodeBlkSize=1048576
03809804-InodeBlkCount=199937
03809804-InodeMT="Wed Aug 10 17:29:41 CEST 2016"
03809804-InodeCT="Tue Jun 21 23:52:18 CEST 2016"
03809804-InodeAT="Wed Aug 10 17:29:41 CEST 2016"
03809804-InodeUid=00000000
03809804-InodeGid=00000180
03809804-InodeMode=00000003
03809804-InodeZLA=00000000
03809804-InodeTBZ=00000000
03809804-InodeCow=00030d01
03809804-InodeBlocks=00000000
03809804-InodeRDM_id=00000000
03809804-InodeContent=00000000
This is for the flat.vmdk:
04c09804-InodeIDdec=79730692
04c09804-InodeIDhex=04c09804
04c09804-InodeID2= 20
04c09804-InodeNLink= 1
04c09804-InodeType= 3
04c09804-InodeType=file
04c09804-InodeFlags= 0
04c09804-InodeSize= 0
04c09804-InodeBlkSize=1048576
04c09804-InodeBlkCount=204800
04c09804-InodeMT="Wed Jun 22 07:43:03 CEST 2016"
04c09804-InodeCT="Wed Jun 22 07:43:03 CEST 2016"
04c09804-InodeAT="Wed Jun 22 08:39:42 CEST 2016"
04c09804-InodeUid=00000000
04c09804-InodeGid=00000180
04c09804-InodeMode=00000003
04c09804-InodeZLA=00000000
04c09804-InodeTBZ=00000000
04c09804-InodeCow=00032000
04c09804-InodeBlocks=00000000
04c09804-InodeRDM_id=00000000
04c09804-InodeContent=00000000
