Just checked: No IP controls are in place, and even the MS iscsi command line returned Authentication Failure on the command "iscsicli qaddtargetportal FQDN"
I've tried by IP, using two machines on different subnets that both could ping the DSS (and vice versa), and FQDN.
I probably disabled the dynamic discovery in VMWare (is it by default? I never checked, since I install by kickstart script and use static discovery there), but I cannot see how that would be the cause the MS discovery authentication failure.
See if you can telnet to the port and get an answer.
There isnt anything on the DSS side that would block it. In fact DSS will broadcast the LUN on each IP assigned to it.
You can try to remove the target and re-add it. This will not remove data, just don't delete the Volume.
I suspect something is blocking traffic in between the initiator and the LUN.
I will make one last attempt before calling for your help: I will shutdown the ESX, reboot the DSS machine so it runs the latest updates and no connections are running and try to connect.
If the cert issue (weird: removing the root from the trusted authorities in my personal store gets me to the login, but no further?) has been fixed, I will download some logs.
Make sure after rebooting the ESX server to use the reset function for the iSCSI Targets in Location
MAINTENANCE -> connections -> Function: iSCSI connection reset.