iSCSI Fables and Open-E Woes!

After too many hours to mention I finally got all the knobs turned in the right positions and got things going. There are a lot of fables in the vmware forums that can lead you astray when working with iSCSI. You really should have a solid understanding of what is expected and be cautious when reading forums. Some of the posts can seem so convincing that you will wind up on a long journey to nowhere!

Notes on iSCSI when using the software initiator

First the vSwitch configuration should be as follows:

Start with a clean slate. I would disable and kill the iscsi configuration with: esxcfg-swiscsi –d and then esxcfg-swiscsi –k.
Setup a virtual switch to support a service console and a vm-kernel.
The ip addresses of both the service console and the vm-kernel should be unique but on the same subnet.

The subnet mask should be correct for the network numbers. Don’t be fooled into believing the vmkernel should be a subnet and not a unique full IP address.
Make sure the target is setup as a replication target (Important for multiple hosts) This has to be selected when the LUN is created. If you don’t, multiple hosts will not be able to see and use the LUN(s).

After the target(s) is configured, re-enable the iscsi process with esxcfg-swiscsi –e and then setup the Discover Targets with vmkiscsi-tool –D –a x.x.x.x vmhba40 (where x.x.x.x is the IP address of your target).

After this returns to the prompt force a rescan with esxcfg-swiscsi –s.

You should see a LUN get added. Confirm this with the command vmkiscsi-tool –l –T vmhba40.

Another thing that is very frustrating is the need to restart the hostd daemons every time you modify a value from the CLI if you expect to see the modifications in the VIC.

The service mgmt-vmware restart command does this for you. But of course this will hose what ever your looking at and you will have to close and reopen the VIC.

You can then add the storage to the host. Once added if the networking is setup and if the target was setup correct, you should be able to see your storage on all hosts.

Added 4-30-07:
Don't use questionable hardware in the iSCSI hardware path. This may seem obvious but a cheap switch or substandard Network cards and/or cables can lead to huge problems. :mad:

Regards,
Do-K

Quote:

---

Originally Posted by Do-K

After too many hours to mention I finally got all the knobs turned in the right positions and got things going. There are a lot of fables in the vmware forums that can lead you astray when working with iSCSI. You really should have a solid understanding of what is expected and be cautious when reading forums. Some of the posts can seem so convincing that you will wind up on a long journey to nowhere!

Notes on iSCSI when using the software initiator

First the vSwitch configuration should be as follows:

Start with a clean slate. I would disable and kill the iscsi configuration with: esxcfg-swiscsi –d and then esxcfg-swiscsi –k.
Setup a virtual switch to support a service console and a vm-kernel.
The ip addresses of both the service console and the vm-kernel should be unique but on the same subnet.

The subnet mask should be correct for the network numbers. Don’t be fooled into believing the vmkernel should be a subnet and not a unique full IP address.
Make sure the target is setup as a replication target (Important for multiple hosts) This has to be selected when the LUN is created. If you don’t, multiple hosts will not be able to see and use the LUN(s).

After the target(s) is configured, re-enable the iscsi process with esxcfg-swiscsi –e and then setup the Discover Targets with vmkiscsi-tool –D –a x.x.x.x vmhba40 (where x.x.x.x is the IP address of your target).

After this returns to the prompt force a rescan with esxcfg-swiscsi –s.

You should see a LUN get added. Confirm this with the command vmkiscsi-tool –l –T vmhba40.

Another thing that is very frustrating is the need to restart the hostd daemons every time you modify a value from the CLI if you expect to see the modifications in the VIC.

The service mgmt-vmware restart command does this for you. But of course this will hose what ever your looking at and you will have to close and reopen the VIC.

You can then add the storage to the host. Once added if the networking is setup and if the target was setup correct, you should be able to see your storage on all hosts.

Added 4-30-07:
Don't use questionable hardware in the iSCSI hardware path. This may seem obvious but a cheap switch or substandard Network cards and/or cables can lead to huge problems. :mad:

Regards,
Do-K

---

Very good stuff!

Some follow-up questions if I may:

If you have multiple NICs in a DSS server, how can I for traffic for a specific target to use that nic? For example:

I will have 4xGE in the server, and I have 4 LV's setup. I want to mount these to a cluster of ESX servers, however I want each NIC in the DSS server to handle traffic for that LV.

So target0 has an IP and I want it to handle traffic for lv000, target1 has a different IP and I want it to hande traffic for lv001, etc.

Is this possible?

I see where you define the eth address, and then the targets, but where do you tell a target to use a specific IP and force traffic for that lv up/down that GE.

I am trying to get more performance out of the box using smaller raid groups, mounted as multiple LV's to the esx cluster.

tjk

Quote:

---

Originally Posted by Do-K

After too many hours to mention I finally got all the knobs turned in the right positions and got things going. There are a lot of fables in the vmware forums that can lead you astray when working with iSCSI. You really should have a solid understanding of what is expected and be cautious when reading forums. Some of the posts can seem so convincing that you will wind up on a long journey to nowhere!

Notes on iSCSI when using the software initiator

First the vSwitch configuration should be as follows:

Start with a clean slate. I would disable and kill the iscsi configuration with: esxcfg-swiscsi –d and then esxcfg-swiscsi –k.
Setup a virtual switch to support a service console and a vm-kernel.
The ip addresses of both the service console and the vm-kernel should be unique but on the same subnet.

The subnet mask should be correct for the network numbers. Don’t be fooled into believing the vmkernel should be a subnet and not a unique full IP address.
Make sure the target is setup as a replication target (Important for multiple hosts) This has to be selected when the LUN is created. If you don’t, multiple hosts will not be able to see and use the LUN(s).

After the target(s) is configured, re-enable the iscsi process with esxcfg-swiscsi –e and then setup the Discover Targets with vmkiscsi-tool –D –a x.x.x.x vmhba40 (where x.x.x.x is the IP address of your target).

After this returns to the prompt force a rescan with esxcfg-swiscsi –s.

You should see a LUN get added. Confirm this with the command vmkiscsi-tool –l –T vmhba40.

Another thing that is very frustrating is the need to restart the hostd daemons every time you modify a value from the CLI if you expect to see the modifications in the VIC.

The service mgmt-vmware restart command does this for you. But of course this will hose what ever your looking at and you will have to close and reopen the VIC.

You can then add the storage to the host. Once added if the networking is setup and if the target was setup correct, you should be able to see your storage on all hosts.

Added 4-30-07:
Don't use questionable hardware in the iSCSI hardware path. This may seem obvious but a cheap switch or substandard Network cards and/or cables can lead to huge problems. :mad:

Regards,
Do-K

---

Also, why do you need to have replication on the LV's enabled for multiple hosts to access them, especially if you are not replicating the volumes?

tjk