You're right. In general, if your storage zoning configuration and presentation of your block layer storage devices (aka LUNs, including RDMs and VMFS volumes) is consistent among hosts in your clusters, and thus the target host has access to them, there won't be any issue with vMotioning VMs with physical or virtual mode RDMs and this is supported.
The only exception to this is if you're running in-guest-clusters with shared storage (like MSCS) on these VMs or other applications that use persistent SCSI reservations.
vMotioning of clustered VMs with shared storage (which are using physical mode RDMs) is not supported by VMware:
http://kb.vmware.com/kb/1037959
You can read more on the technical background about this here:
But if you're not using clustering or something similar with the RDMs on the VMs, you're good to go.