OK, so I didn't get any suggestions. I looked with some Linux experts that could not explain why I could not delete directly from file system either.
So, i formatted my disks, installed the latest FreeNas 8.3.0 and this time with no RAID Z.
I created my volumes and map them to ESXi and it is all working well now. Probably even 5 times faster.
So I am not sure what the problem was. If it was the RAID Z or the version of FreeNas but all working now.