Have you recently moved any VMs off that datastore? Maybe this reduces the VMs that need to be checked!?
Btw. what's the exact error message that you get? Just want to rule out that it's a HA heartbeat datastore that prevents you from removing the datastore.
André