In case anyone reads this, I raised a case with Dell and a guy there with SAN and vmware knowledge fixed it. He made 2 changes.
The first change, which I think was the one which fixed it, was to set the MTU. I'd ticked "jumbo frames" on the SAN, and set the MTU on the port group to 9000. I hadn't set the MTU to 9000 on the vSwitch, didn't realise it was necessary. The Dell guy did this, and afterwards the host could ping the SAN with large packets and mutual CHAP worked. We ticked the check boxes to use the initiator name in the CHAP dialog box.
The second change surprised me, he removed the vmkernels from the Network Configuration tab on the iSCSI vmhba. He said that this was necessary for our kind of SAN. I'll do some more reading about this, this is our first iSCSI SAN but we certainly now have CHAP authenticated paths between host and SAN.