Good day,
What's your NetApp hardware config with respect to disks, aggregates, and controllers? Are there other workloads on these controllers or ESXi hosts?
For VMware on NetApp, there are a few NFS settings that NetApp recommends for your ESXi hosts. If you're using the Virtual Storage Console, you can do this somewhat automatically with a reboot to finish the config change. You don't have to use VSC, though. You can change this parameters manually using the VSC documentation: https://library.netapp.com/ecmdocs/ECMP1392339/html/GUID-346ACB95-6AD4-4DEA-8901-C9697AC3530F.html
Also check out this NetApp KB for how to configure vSphere 5.x for Data ONTAP 8.x: https://kb.netapp.com/support/index?page=content&id=1013275
Is Storage I/O Control enabled? If so, check out this post and it's weird resolution: http://blog.igics.com/2013/09/high-latency-on-vsphere-datastore.html
Some immediate and obvious things to check include:
If you're using jumbo frames, ensure each device in the path is configured correctly for them, from vmkernel ports to switch ports to NetApp interfaces. If using jumbo frames, check out this KB VMware KB: Troubleshooting disk latency when using Jumbo Frames with iSCSI or NFS datastores
Look for PAUSE frames being sent or received from devices in the path. This is an indication of a device becoming saturated and not being able to keep up with sent data. Think Wireshark here.
All the best,
Mike
-----------------------------------------
Please consider marking this answer "correct" or "helpful" if you found it useful.
Mike Brown
VMware, Cisco Data Center, and NetApp dude
Consulting Engineer
Twitter: @VirtuallyMikeB
Blog: http://VirtuallyMikeBrown.com
LinkedIn: http://LinkedIn.com/in/michaelbbrown