vSphere 4 - Support on the Way?

[1parkplace]

Response from someone in the VMWare community regarding my uploaded VMKernel Logs:
__________________________________________________ _______
There seems to be an issue with storage:


Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.023 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba33:CH:0 T:1 L:2 : Abort task response indicates task with itt=0x1107006 has been completed on the target but the task response has not arrived ...
Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.272 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_ConnSetupScsiResp: vmhba33:CH:0 T:1 CN:0: Task not found: itt 17854470


17854470(dec)=110700(hex)


1- There is an IO timeout (i.e storage is not responding to IO on time) which cases ESX iSCSI initiator to send an abort for that IO.


2- It appears that the storage responds to that with "task does not exist" but later the storage sends response the IO task. That is in violation of iSCSI protocol and ESX initiator drops the connection. This seems to keep happening very often.


ESX 3.5 s/w iscsi initator would just ignore that case but ESX 4 initiaror is very strict about protocol violation.


It appears you are using Open-E DSS, I do not think it is certified with ESX4 yet. Could you post the version of DSS you are using ?.

__________________________________________________ ____

[To-M]

Thanks for the looking into this - I have forwarded this to engineers to verify. We are watching this thread. Also send me new logs once the changes have been made from the tuning options.

[1parkplace]

Tuning Configuration is complete.

New log packages sent to your email.

Thanks!

Drew

[cschiff]

Drew,

We too are facing a similar scenario; however we have not upgraded our ESX environment to vSphere 4.0 as I noticed they were not on the HCL for vSphere 4.0 Storage. However I really want to be able to upgrade to vSphere 4.0 so I am hoping they can resolve your issue and get recertified. However, I was not to optimistic on the speed of this happening as I was told by support that they may look at testing and recertifying sometime later this year, but as of yet no testing had been done with vSphere 4.0. I am going to continue to monitor your thread for progress resolving this issue. Hopefully since you already on vSphere 4 and need DSS to be compatible it doesn’t take until the end of the year to get that.

[cschiff]

Drew,

I am also curious what version of DSS you are running?

Were you experiencing any iSCSI timeout errors when using ESX 3.5?

What RAID level are you running on your DSS SAN?

We had fought iSCSI timeout issues with DSS for many months. I tried configuring the DSS SAN with all of the recommended perfomance settings as specified by Open-e support. Things got a little better, but when the SAN was under moderate to heavy load we would still get CMD Abort and Task Not Found errors. What I found to be our ultimate solution was to rebuild our RAID set as RAID 10. Previously I was using RAID 5 and had my VMs split accross 2 DSS SANs, so each was had 50% of the ESX load. Now with the SANs at RAID 10, I am able to run ALL of my VMs on 1 SAN with better performance and NO iSCSI timeouts or errors, under any load.

[1parkplace]

I'm running the 3513 build of DSS. I have two servers which I run DSS on, each server has 3 RAID configs and I map their LUNs to a single target. I have, on each server, different RAID setups ranging from RAID1, RAID5, RAID 10.

I also run these targets through a 10GbE Network, so maybe this is why I never ran into timeouts, are you using a similar setup?

On ESX3.5u3 I was not getting this same issue... however, I was experiencing finicky connection drops due to the use of the NetXen 10GbE NICs in my systems.

This weekend I will be installing Intel 2xCX4 10GbE NICs in my machines to alleviate that problem.

[1parkplace]

my VMKernel logs have been fairly clean since I changed my iSCSI Daemon tuning options to what Todd recommended.

I am still getting these messages though in my VMKernel logs on ESX4.0:

Jun 5 11:08:57 vmhost-1 vmkernel: 6:01:50:02.470 cpu6:5337)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported

[cschiff]

We are running DSS Build 3511 with ESX 3.5 U4. I have 2 DSS SANs each configured the same with 1 RAID 10 volume. Each SAN has 6 x 1 Gbps NIC ports. 2 are Bonded using Round-Robin for the GUI and 4 are Bonded using Round-Robin for iSCSI. These SANs only have 1 iSCSI target with multiple LUNs, each exposed only to a 3 ESX host Cluster.

[masim]

Hello,

I had similar problem, and it was a hardware issue. I tried changed the IETD iSCSI Target and it gets better.

Response from someone in the VMWare community regarding my uploaded VMKernel Logs:
__________________________________________________ _______
There seems to be an issue with storage:


Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.023 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba33:CH:0 T:1 L:2 : Abort task response indicates task with itt=0x1107006 has been completed on the target but the task response has not arrived ...
Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.272 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_ConnSetupScsiResp: vmhba33:CH:0 T:1 CN:0: Task not found: itt 17854470


17854470(dec)=110700(hex)


1- There is an IO timeout (i.e storage is not responding to IO on time) which cases ESX iSCSI initiator to send an abort for that IO.


2- It appears that the storage responds to that with "task does not exist" but later the storage sends response the IO task. That is in violation of iSCSI protocol and ESX initiator drops the connection. This seems to keep happening very often.


ESX 3.5 s/w iscsi initator would just ignore that case but ESX 4 initiaror is very strict about protocol violation.


It appears you are using Open-E DSS, I do not think it is certified with ESX4 yet. Could you post the version of DSS you are using ?.

__________________________________________________ ____

[jsinclair]

Hello,

I had similar problem, and it was a hardware issue. I tried changed the IETD iSCSI Target and it gets better.

Do you mean you changed from SCST to IETd? Or from IETd to SCST ?? And what got better? I am using SCST and I am not having the timeout issues, just slow performance.

Todd -
I can try and change to balance-alb, but I think in the past I have experienced "port-flapping" on our Cisco switches where it gets confused on which interface to use, but I can certainly give it a shot. Do we maybe want to try some of those target settings you suggested initially? I will also forward the logs from today running with balance-rr.

Thanks again,
Jason