OK, after working with both NetApp and VMware on a conference call (after both pointed to each other) the bottom line "fix" was to remove the iSCSI port bindings, delete the broken paths, delete the vSwitches then rebuild them again.
1. Create vKernel vSwitch with proper port group
2. Set the MTU to 9000 on both the Port Group and vSwitch
3. Create the iSCSI Port Bindings for each interface
4. Re-scan
5. Successfully create the Datastore
Strangely enough, I had to repeat this process 3 times for one of the blades, but it worked the first time for the other 3 blades involved.
VMware Tech Support did provide the following as one look at the LUN's from the CLI:
1. cd /vmfs/devices/disks
ls -l
we see and output of all available disks
Identify the Naa.xxxxxxxx of Lun
2. Run this for Naa.xxxxx reported in step 2
partedUtil getptbl /vmfs/devices/disks/naa.xxxxxxx
if there is a partition you might see something similar to
1 63 2249099 131 128 > 1 is partiton number
or
you might see and error, if error please contact tech support
3. If there is partition you can delete
partedUtil delete /vmfs/devices/disks/naa.xxxxxxx 1
NOTE: This process did not work, but is still valuable nevertheless. It could have worked if the LUN had been used by another operating system and needed cleaning up. But my situation was all greenfield.