Note: Discussion successfully moved from VMware ESXi 5 to VMware vSphere™ Storage
The performance of a in-house built storage solution will depend on a variety of factors, including disks, cache, etc. The big thing here is the support. If you build your own server, make sure the software vendor is listed in the VMware HCL as supported. I have also had good luck with NFS, running on CentOS for smaller and lower IO VMs. Of course, this isn't supported either. Also, keep your backups in mind.