I have got my ESX 3.5 server to connect via iSCSI.
So here is what I did.
DSS-Lite
Create the iSCSI target
Make sure that your lun is not set to zero. ESX does not like lun0.
You need to initialize the volume. You can not do this with ESX. So I connected a Vista Workstation to the iSCSI target and went to storage manager and initialize the volume.
Do not format the volume.