20ms for an RDM on an empty storage seems a bit high - you really need to check what the ESX is measuring. Use KB http://kb.vmware.com/kb/1008205 to check performance.
Usually, under normal operations with no issues, I see 3-10ms latency on VMFS volumes (with around 10VMs each)... this is an average of various arrays I have already analyzed for performance. This obviously depends on the SCSI protocol, disk type (SAS/SATA/FC), RAID, etc.