Mike, Thank you for the very thorough and detailed explanation.
In terms of Netapp controllers there will be 2240s, 3240s and 6240s. The controller type is dependent on the size of the site. The filers will not only support the virtual infrastructure and also house other data. Links per controller are two minimum. Sites will have dual controllers (I haven't seen a site survey to make sure it's the case for all sites, I am working under that risky assumption which I will hopefully clarify soon). Even though I will have different controller types I am looking to standarize the design as much as possible since all sites will be supporting the same size virtual infrastructure.
I like your approach of using the 4 links with one vmkernel port. I could just make a 4 port port-channel/LACP or MLACP depending on the amount of switches I'll have. Yes, I was thinking of using dynamic multimode VIFs and have 1:1, datastore:node ip set.
I have to make this infrastructure scalable but at first roll out I will have two datastores to work with. Which kind of puts me in a spot of not maximizing on uplink usage.
Based on your guidance could I assume if I only have two datastores there will only be using two uplinks in use if I use a multimode VIF and one IP per datastores? The other uplinks will just be taking a role in the LACP port-channel.
Thanks again for the very valuable input.