Due to having only a single nic on my NAS I have done the following to allow easy management of the NAS's web GUI over my LAN and still separate the NFS subnet from vsphere.
I have a vsphere environment on subnet 192.168.1.0/24. The default gateway of all my ESXi servers is 192.168.1.1, this the interface of my router dedicated to the ESXi hosts.
I have a NAS with only 1 nic. I have given this NAS an IP 192.168.2.2/24. My Router also has an interface configured as 192.168.2.1/24, this subnet is dedicated to NFS shares.
My PC's are on subnet 192.168.3.0/24 with default gateway 192.168.3.1, this is the interface of my router. To manage the NAS from my PC I can simply navigate to 192.168.2.2 and because my router has an interface in that subnet it routes my http request as such.
In each of the ESXi hosts I have added vmk1 with an IP in the 192.168.2.0/24 subnet and the default gateway (from vmk0) of 192.168.1.1. From what I understand when adding an NFS share ESXi will first check to see if any vmk's are on the same subnet as the NFS server and if so send the request via layer 2. If no vmk's are on the same subnet as the NFS server then requests are sent to the default gateway via the earliest added vmk on the host e.g vmk0
I presume these rules continue to apply after mounting the NFS share but just in case I have firewalled all ESXi host requests to the 192.168.2.0/24 subnet in my router making sure the only communication with the NAS can be over layer 2.
Does this all sound acceptable?
Cheers.