Hi,
Your VMs will not have to be powered off when you do an in place upgrade from VMFS3 to VMFS5. It's done on the fly, there are however some caveats. Please read this whitepaper about the considerations on upgrading to VMFS5.
Imo, I'll go for creating a new VMFS5 datastore in stead of upgrading existing VMFS3 to fully take advantage of the new features. Simply move the VMs to a temporary LUN, delete the old one, create a new one formatted with VMFS5. Move the VMs back.
RDMs are indeed not affected.
Hope this helps.
Regards.