I/O Errors with iSCSI

Dear all,

We are currently planning on purchasing open-e DSS for our setup. During our evaluation we're running into a problem executing a larger I/O load on the iSCSI disk provided by open-e:

I/O Errors detected on unit S001. To prevent data loss it is recommended to shut down the system. Please contact your hard drive vendor for further support.

The environment is the following: we have a Dell server which runs ESXi 5.1. On this server there is exactly one VM: a fresh open-e DSS V7 install. The ESXi provides a 400GB large disk to the VM (the disk is placed on a RAID1 controlled by Dell's R710 controller, backed by 2 SATA disks). Open-E then provides an iSCSI target which is mounted on ESXi.

Now we wanted first to convert some VMs we have running on desktop machines by executing vmkfstools via the shell on ESXi. The source and destination .vmdk files reside both on the iSCSI-mounted disk. When executing the command we pretty quickly get the I/O errors message as above. This is reproducible.

We are happy to help troubleshooting, please let us know which information you need.

The ESXi vmkwarning log is:

2012-11-05T20:43:50.473Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnCommandResponse: SCSI response overflow: residual 1048064
2012-11-05T20:43:50.473Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnCommandResponse: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:43:50.473Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnCommandResponse: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:44:00.659Z cpu12:19502)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.6e6d306e435a784c" state in doubt; requested fast path state update...
2012-11-05T20:44:04.657Z cpu12:19502)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "eui.6e6d306e435a784c" state in doubt; requested fast path state update...
2012-11-05T20:44:14.730Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194275 (refITT=0x19426d) timed out.
2012-11-05T20:44:24.732Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194277 (refITT=0x19426e) timed out.
2012-11-05T20:44:29.053Z cpu2:22911)WARNING: UserLinux: 1331: unsupported: (void)
2012-11-05T20:44:34.733Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194279 (refITT=0x19426f) timed out.
2012-11-05T20:44:44.735Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19427a (refITT=0x194270) timed out.
2012-11-05T20:44:54.737Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19427c (refITT=0x194271) timed out.
2012-11-05T20:45:04.739Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19427e (refITT=0x194272) timed out.
2012-11-05T20:45:14.741Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19427f (refITT=0x194273) timed out.
2012-11-05T20:45:24.743Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194281 (refITT=0x194274) timed out.
2012-11-05T20:45:39.843Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194283 (refITT=0x19426d) timed out.
2012-11-05T20:45:49.845Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194284 (refITT=0x19426e) timed out.
2012-11-05T20:45:59.847Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194286 (refITT=0x19426f) timed out.
2012-11-05T20:46:09.849Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194288 (refITT=0x194270) timed out.
2012-11-05T20:46:19.851Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194289 (refITT=0x194271) timed out.
2012-11-05T20:46:29.853Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19428b (refITT=0x194272) timed out.
2012-11-05T20:46:39.855Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19428d (refITT=0x194273) timed out.
2012-11-05T20:46:49.857Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x19428e (refITT=0x194274) timed out.
2012-11-05T20:46:59.859Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194290 (refITT=0x19426d) timed out.
2012-11-05T20:47:09.861Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194291 (refITT=0x19426e) timed out.
2012-11-05T20:47:19.862Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194293 (refITT=0x19426f) timed out.
2012-11-05T20:47:29.864Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194295 (refITT=0x194270) timed out.
2012-11-05T20:47:39.866Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:0 L:0 : Task mgmt "Abort Task" with itt=0x194296 (refITT=0x194271) timed out.
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x19427e
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x19428b
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x194275
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x194283
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x194290
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x19427c
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x194289
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Conn [CID: 0 L: 10.20.1.101:52641 R: 10.20.1.102:3260]
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: vmhba37:CH:0 T:0 CN:0: TMF Response PDU: Referenced task not found: itt 0x194296
2012-11-05T20:47:45.155Z cpu9:8816)WARNING: iscsi_vmk: iscsivmk_ConnSetupTMFResp: Sess [ISID: 00023d000002 TARGET: iqn.2012-11:dss1.target0 TPGT: 1 TSIH: 0]