vSphere 4 - Support on the Way?

Jason - Nice details on the graphics!!

I have forward this onto the engineers that are involved with DSS V6.

Logs from what I saw did not show any issues with packet loss only some target settings that I may want you to test with and change on our end from the console in CTRL ALT W then Tuning Options then iSCSI daemon settings and then select the target and try these settings. It will do a reset on the VMware side.

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

Todd -
Thanks! I just find it easier to visualize something than write it down :-)

I tried the settings you suggested, and it lost my iscsi connections on all vmware ESX servers (2 x esxi 3.5, and 4 x vsphere 4). I was unable to get them restored, even after a reboot of an ESX server, vCenter server, and SAN1. I had to set everything back to the default settings to make everything work again.

Any idea what I may have done wrong?

Thanks again,
Jason

Engineers would like to see if you can test with the IETd instead of the default SCST - to change this go to the console then enter CTRL + ALT + W then Tuning options then iSCSI daemon then iSCSI solution then select IETd - you will lose connections from a reset and you will need to reconnect from the initiator side.

Also can you try without the bond as well. Then send the results with logs to support@open-e.com in the subject line of the email please copy and past this

Re: [Ticket#1009585] DSS V6 tests are slow with VMware ESX 3.5 and 4.0

We have a team watching this ticket.

Todd,
No problem. I can change to IETd, but since it is during business hours, I may have to wait. If I failover to san2, change san1 to IETd and then failback should that help prevent the downtime?

Also - since the bond is the main iscsi interface which has the virtual IP assigned to it, how will this affect my setup? Should I try a different bonding method?

Thanks,
Jason

Sorry for the complications as I see this will be more complicated :(

Thinking this thru you will have to stop the connection because you will eventually be stuck with resetting the connection from the vm side because. We have not tested this type of switch over - sorry for that. It may work that you can stop the Primary server then switch then enable again then failback then stop the secondary then switch back on again.

Now for the bond you will have to stop the service all together, again sorry about that. I would do this last and test again.

Todd -
What I will try is to switch over to IETd tonight (actually in the next couple hours). I will force a failover to SAN2, enable IETd on SAN1, failback and then enable IETd on SAN2.

Hopefully this should be a seamless transition. Once moved over, do you want me to let it run? Or try those settings you suggested yesterday on the target side?

As far as breaking the bond, that will be the tricky part and I would like to wait to do that last.

Also, I have been doing some basic disk testing with CrystalDiskMark (Win) and hdparm (Linux), and I have noticed the volume group on the external PERC controller (RAID6 8x450gb 15k SAS) gets me an average of ~110MB/s Seq Read, and ~65MB/s Seq write in Windows and a buffered disk read of ~180MB/sec in Red Hat Enterprise 5 x64 (hdparm -tT /dev/sda).

Now, the internal PERC controller (RAID10 6x450gb 15k SAS) gets me a sluggish Seq read of ~50MB/s and a Seq write of ~30MB/s and as low as 5MB/sec. If I do a Storage vMotion to the External PERC, it gets a little better at ~95MB seq read and ~33MB seq write. I'm confused...all controllers have the same setup except the PERC 5/e in SAN2 (refer to my diagram for reference).

Any thoughts? I thought I had some consistent results, but a couple of the servers that were dropping is on the external array. Doesn't matter where I move them to. If I've confused you, let me know and I might be able to explain better.

Thanks again for all your help. I will update the IETd switchover later tonight...

Jason

Sorry for not getting back fast enough - very busy times :).

Just let it run after the switch over - let the bond wait so we can trouble shoot one at a time.

Concerning the performance is was ready to jump on the disk drives but now in the end of your message that squelched that idea, unless firmware and or using 32bit mode made the difference of the drivers on the system - but don't think that is the case. We would have to see the logs to see if there is anything we can find to point us in the right direction. I would think the RAID 10 would do very well.

Todd -
Yes, that's what I thought. I have updated all firmware on both systems, and all RAID controllers, system, and bios are up to date. Now, the only difference in controllers is the PERC 6/e has 512MB (RAID 6 array) and all the others have 256MB.

Now, I failed over to SAN2, changed the iSCSI from SCST to IETd on SAN1 and failed back. That all went ok...well....almost.

All the ESX servers lost connections to the datastores on the iscsi volumes. After doing a rescan, the disks show back up but do not have a datastore assigned to it. I can goto Add Datastore and see the LUNS, but this would require them to be reformatted..

I changed back to SCST to get everything back up. If you see this in time, let me know if you want me to give anything else a shot. I can have temporary interruptions for the next few hours. I can be reached via phone if necessary (PM me for info) and can give you access to shadow me, etc.

Let me know.

Thanks,
Jason

I sent you a message - give me your number so i can call you.

Done.

Thanks!

(Accidentally posted this in the wrong thread...)


Todd -
Thanks again for taking time last night to help me out. I have done what you suggested and killed my bond, changed the IP of the iscsi interface to a different subnet, and then setup the virtual IP again.

Now, what I noticed was the speed tests were a little more consistent with each other after about 20 minutes (a lot of traffic initially because the VM's were down for about an hour). Of course, if I ran more than one test at a time the gig iscsi interface on the SAN couldn't keep up. I was seeing an average of ~105MB/s seq read, ~55MB/s seq write on both the RAID6 external lun and the RAID10 internal LUN.

I then created a balance-rr bond with all 4 of the Intel GIG iscsi interfaces. The tests were a little better than the 802.3ad, but not as "bursty." Once again, the speeds vary from lun to lun and when they are run. I have seen the low 90's on the read and even in the 70's on the writes. I think were getting closer. I will let it run more during the day before sending some logs. I have uploaded the logs from both servers when it was in a single, no bond, gig interface.

I'll keep you posted and possibly get some performance numbers once I can find some consistency.

http://upload.vmind.com/web-pub/cera...6-18_00-43.zip

Thanks again!

Jason

Jason - Thanks for the update, seems we are getting closer. The bonds will start kicking in when there are many systems or requests coming then it will do better. You can also try the ALB bond. I heard that is doing well. Looks like we will work on the 802.3ad now.

Hello,

I had similar problem, and it was a hardware issue. I tried changed the IETD iSCSI Target and it gets better.

Quote:

---

Originally Posted by 1parkplace

Response from someone in the VMWare community regarding my uploaded VMKernel Logs:
__________________________________________________ _______
There seems to be an issue with storage:


Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.023 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba33:CH:0 T:1 L:2 : Abort task response indicates task with itt=0x1107006 has been completed on the target but the task response has not arrived ...
Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.272 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_ConnSetupScsiResp: vmhba33:CH:0 T:1 CN:0: Task not found: itt 17854470


17854470(dec)=110700(hex)


1- There is an IO timeout (i.e storage is not responding to IO on time) which cases ESX iSCSI initiator to send an abort for that IO.


2- It appears that the storage responds to that with "task does not exist" but later the storage sends response the IO task. That is in violation of iSCSI protocol and ESX initiator drops the connection. This seems to keep happening very often.


ESX 3.5 s/w iscsi initator would just ignore that case but ESX 4 initiaror is very strict about protocol violation.


It appears you are using Open-E DSS, I do not think it is certified with ESX4 yet. Could you post the version of DSS you are using ?.

__________________________________________________ ____

---

Quote:

---

Originally Posted by masim

Hello,

I had similar problem, and it was a hardware issue. I tried changed the IETD iSCSI Target and it gets better.

---

Do you mean you changed from SCST to IETd? Or from IETd to SCST ?? And what got better? I am using SCST and I am not having the timeout issues, just slow performance.

Todd -
I can try and change to balance-alb, but I think in the past I have experienced "port-flapping" on our Cisco switches where it gets confused on which interface to use, but I can certainly give it a shot. Do we maybe want to try some of those target settings you suggested initially? I will also forward the logs from today running with balance-rr.

Thanks again,
Jason

Jason - You are correct let's go with the other settings and abort the bond change. Thanks for the update.

Quote:

---

Originally Posted by To-M

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

---

Todd -
I tried these settings again on my V6 beta SAN's, and once again all ESX servers (3.5 and 4) couldn't reconnect. I then tried to set them one at a time to see which setting caused the problem, and it's the MaxBurstLength setting. Every other setting allows the initiators to reconnect. Any other value I should try? Also, I have the logs from the balance-rr and you can download them below.

Thanks,
Jason

http://upload.vmind.com/web-pub/cera...6-18_21_16.zip

Engineers are investigating the bond issue as this is where they think the problem is also they are looking into the maxburst issue as well. I would leave the defaults for now. They will have to get some update for V6 to solve this issue with SCST.

Todd -
Sounds good. I will wait for advice then. It seems to be running OK now and not dropping machines, but the performance is still a little less to be desired.

If you need anything else from me let me know.

Thanks again for your help.

Jason

Todd -
I have been running with the v6 beta (update1) in the balance-rr mode since we last spoke, and the virtual machines are no longer dropping in vSphere 4. The performace is still a little on the middle tier, but at lease I have the stability problems resolved. I am getting on average 50-60MB/s read and 40-50MB/s write on both the RAID10 and RAID6 LUNS within VMware.

Is there anything else you would like me to try or change? I can also upload logs from both systems if you would like. I have roughly 38 days left on the beta demo. Any word on the final release?

Thanks again,
Jason

Hey Guys,

I think I have my issues resolved as well. Our issue turned out to be the NetXen-based 10GbE NICs. They were HP NC510C cards (Single Port CX4).

Once these cards were replaced with Intel 82598EB-based cards by SuperMicro the LUN dropping has disappeared and all VMKernel logs and DSS logs look clean and clear.

I will keep you guys posted with test results once I get my balance-rr setup on the 2-port NICs.

Thanks for the update. Engineers will compare other results and look to see if there is anything else. We are working with the SMC 10GbE nics but we wont have time to get the drivers ready for this version.

Todd - Do you want me to send my logs?

Please do! I will be gone and wont able to review them until next week.

Todd -
Thanks again. Here they are:
http://upload.vmind.com/web-pub/cera...-June30.tar.gz
http://upload.vmind.com/web-pub/cera...-June30.tar.gz

Let me know if you need anything else.

Jason

Engineers informed me that the new official version has resolved those errors that you had earlier and try test with the latest version and you should be good to go now.

iscsi-scst: ***ERROR*** Connection with initiator
iqn.1998-01.com.vmware:ceraesx6 unexpectedly closed!
iscsi-scst: ***ERROR*** Connection with initiator
iqn.1998-01.com.vmware:ceraesx6 unexpectedly closed!
iscsi-scst: ***ERROR*** Connection with initiator
iqn.1998-01.com.vmware:ceraesx6 unexpectedly closed!

So, is upd 4 the official v6 release? or is it still beta and the official release is comming?

Todd -
Thanks for checking on this. I have some questions regarding installing the V6 final release. Mostly regarding the beta trial. I have roughly 30+ days left on on the beta update 1 trial, and was wondering what steps I should take to test out the full version of V6.

Do I update the 2nd SAN first, failover and then update the first?
Can I update the beta version, or do I need to do a reinstall?
Will the trial stay at the remaining 30 days, will it reset to 60, or do I need to purchase the licenses?

Let me know what you recommend, as I can do some system maintenance this weekend. We have migrated more physical to virtual in vsphere 4, so it becomes more important to have minor interruptions but also the stability and performance is key. I can also keep assisting in your vSphere 4 testing and certification if you would like. I have some other supermicro systems with 3ware controllers and 16 SATA drives if we want to test out other non-Dell configurations.

Anyways...thanks again and look forward to your recommendations.

Jason

Quote:

---

Originally Posted by itr

So, is upd 4 the official v6 release? or is it still beta and the official release is comming?

---

dss6.0up04.oe_i.zip is the latest DSS V6 version.

Quote:

---

Originally Posted by jsinclair

Todd -
Thanks for checking on this. I have some questions regarding installing the V6 final release. Mostly regarding the beta trial. I have roughly 30+ days left on on the beta update 1 trial, and was wondering what steps I should take to test out the full version of V6.

Do I update the 2nd SAN first, failover and then update the first?
Can I update the beta version, or do I need to do a reinstall?
Will the trial stay at the remaining 30 days, will it reset to 60, or do I need to purchase the licenses?
Jason

---

Hey Todd -
Can you point me in the right direction on this? I'm anxious to move off the beta but want to make sure I have a plan setup first :-)

Thanks again,
Jason

I am so sorry for not getting back to earlier - I was on a much needed vacation. Any way feel free if you want to buzz me or send me an email. Other then that upgrades will be avail. at the end of the month. We have to get a the prod. key ready. We wanted to do the full version first before upgrades. Only thing to do is save your settings and download logs for a backup.

Todd -
Totally understand. I am actually getting ready for a vacation myself :-)

I'll shoot you over an email then.

Thanks again!

Jason

Thanks man!

BTW, re-enabling WB caching has brought back the elusive LUN dropping.

I will be sending another log bundle for you to look at today.

Thanks

We have just moved to ESX4 and not found any problems as yet
2 ESX servers running 5 Vm's and counting
only niggle i can see is that large file copies from a vm guest to and iscsi target are not so fast.

just trying to setup a new iscsi san with open-e dss. we're evaluating different softwares for iscsi and want to use open-e dss.

i have a few questions:

vSphere 4 Server with 1x Quad Port Intel
Open-E DSS Server with 1x Quad Port Intel (last Update, installed today)
Dell 6224 Switch

i just setup a lacp (channel-group 1 mode auto) for the esx4 and a lacp (channel-group 2 mode auto) for the dss.
in dss i choose "new 802.3ad" for the bonding mode.

How do i configure the vSwitch? is there something like a screenshot? i just tried "route based on ip hash", but the performance is very bad.. something like 3mb/s read.

Do i have to setup a hashing mode in the switch?

please help

Can you provide some more detailed information as to how your networking is set up? For example, do you have a vswitch set up just for the vkernel iscsi? Does it have two physical ports of your quad-port NIC? Are you using two ports on your DSS, or more?

My recommendation would be to start out with nothing fancy, no bonding on the vmware or dss or switch. Just establish one path from vmware through the switch to the dss and make sure you have reliable iscsi traffic flowing through there. You should be able to get a raw io of at least 80-100 MB/s on a gigabit port (the theoretical maximum is 125 MB/s). Even if you bond two ports, you probably won't ever get over 120 MB/s from one vmware iSCSI initiator to one dss, since the packets still just go over one physical connection at a time. Bonding connections works great for creating redundant paths but won't give you more bandwidth. (networking experts, please correct me if I'm wrong on this).

So, once you've got your iscsi pipe working, you can turn set up your bonds.

Just my two cents...

For sure we've set a vswitch just for the iSCSI Traffic. We're using 4 NICs for iSCSI@ESX 4NICs for iSCSI@DSS and 1 for Management@DSS+ESX)

How to get more than the theoretical maximum of 125mb/s into one vm? MPIO? How to setup ESX then? Every iSCSI Nic is one vSwitch? I tried this already. got 3mb/s read :)

Hi,

We seem to be experiencing the same issues at a customers site. vSphere 4.0 and DSS V5 is commong among the other replies here. I have opened a support request for this.

Our reseller provided us with DSS5 even tho we got the unit in August, when DSS6 should have been released. Is there any way that we can get the upgrade for DSS5?

Is DSS5 or DSS6 better when using vSphere 4.0?

/Andreas

DSS6 works very well with vSphere4. I have been running solid on DSS6 for 2-3 weeks now, which I was not able to do with DSS5.

I still get connectivity drops on various LUNs under really high I/O with v6, but they recover within seconds... this is a major improvement over v5 where the LUNs would actually have to be dropped and recreated in order for the ESX to pick them up again.

SCST seems to be where the problem was fixed. IETd seemed to be the root of the problem with v5.

This is correct where the new DSS V6 has the SCST and the DSS V5 is using the IET and was certified with VMware 3.5. We are certifying the DSS V6 with VMware 4.0 but not with DSS V5. I would contact your reseller to inquire about the upgrade to DSS V6 to use the SCST.