After too many hours to mention I finally got all the knobs turned in the right positions and got things going. There are a lot of fables in the vmware forums that can lead you astray when working with iSCSI. You really should have a solid understanding of what is expected and be cautious when reading forums. Some of the posts can seem so convincing that you will wind up on a long journey to nowhere!

Notes on iSCSI when using the software initiator

First the vSwitch configuration should be as follows:

Start with a clean slate. I would disable and kill the iscsi configuration with: esxcfg-swiscsi –d and then esxcfg-swiscsi –k.
Setup a virtual switch to support a service console and a vm-kernel.
The ip addresses of both the service console and the vm-kernel should be unique but on the same subnet.

The subnet mask should be correct for the network numbers. Don’t be fooled into believing the vmkernel should be a subnet and not a unique full IP address.
Make sure the target is setup as a replication target (Important for multiple hosts) This has to be selected when the LUN is created. If you don’t, multiple hosts will not be able to see and use the LUN(s).

After the target(s) is configured, re-enable the iscsi process with esxcfg-swiscsi –e and then setup the Discover Targets with vmkiscsi-tool –D –a x.x.x.x vmhba40 (where x.x.x.x is the IP address of your target).

After this returns to the prompt force a rescan with esxcfg-swiscsi –s.

You should see a LUN get added. Confirm this with the command vmkiscsi-tool –l –T vmhba40.

Another thing that is very frustrating is the need to restart the hostd daemons every time you modify a value from the CLI if you expect to see the modifications in the VIC.

The service mgmt-vmware restart command does this for you. But of course this will hose what ever your looking at and you will have to close and reopen the VIC.

You can then add the storage to the host. Once added if the networking is setup and if the target was setup correct, you should be able to see your storage on all hosts.

Added 4-30-07:
Don't use questionable hardware in the iSCSI hardware path. This may seem obvious but a cheap switch or substandard Network cards and/or cables can lead to huge problems.

Regards,
Do-K