vSphere 4 - Support on the Way?

[To-M]

Jason - Nice details on the graphics!!

I have forward this onto the engineers that are involved with DSS V6.

Logs from what I saw did not show any issues with packet loss only some target settings that I may want you to test with and change on our end from the console in CTRL ALT W then Tuning Options then iSCSI daemon settings and then select the target and try these settings. It will do a reset on the VMware side.

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

[jsinclair]

Todd -
Thanks! I just find it easier to visualize something than write it down :-)

I tried the settings you suggested, and it lost my iscsi connections on all vmware ESX servers (2 x esxi 3.5, and 4 x vsphere 4). I was unable to get them restored, even after a reboot of an ESX server, vCenter server, and SAN1. I had to set everything back to the default settings to make everything work again.

Any idea what I may have done wrong?

Thanks again,
Jason

[To-M]

Engineers would like to see if you can test with the IETd instead of the default SCST - to change this go to the console then enter CTRL + ALT + W then Tuning options then iSCSI daemon then iSCSI solution then select IETd - you will lose connections from a reset and you will need to reconnect from the initiator side.

Also can you try without the bond as well. Then send the results with logs to support@open-e.com in the subject line of the email please copy and past this

Re: [Ticket#1009585] DSS V6 tests are slow with VMware ESX 3.5 and 4.0

We have a team watching this ticket.

[jsinclair]

Todd,
No problem. I can change to IETd, but since it is during business hours, I may have to wait. If I failover to san2, change san1 to IETd and then failback should that help prevent the downtime?

Also - since the bond is the main iscsi interface which has the virtual IP assigned to it, how will this affect my setup? Should I try a different bonding method?

Thanks,
Jason

[To-M]

Sorry for the complications as I see this will be more complicated

Thinking this thru you will have to stop the connection because you will eventually be stuck with resetting the connection from the vm side because. We have not tested this type of switch over - sorry for that. It may work that you can stop the Primary server then switch then enable again then failback then stop the secondary then switch back on again.

Now for the bond you will have to stop the service all together, again sorry about that. I would do this last and test again.

[jsinclair]

Todd -
What I will try is to switch over to IETd tonight (actually in the next couple hours). I will force a failover to SAN2, enable IETd on SAN1, failback and then enable IETd on SAN2.

Hopefully this should be a seamless transition. Once moved over, do you want me to let it run? Or try those settings you suggested yesterday on the target side?

As far as breaking the bond, that will be the tricky part and I would like to wait to do that last.

Also, I have been doing some basic disk testing with CrystalDiskMark (Win) and hdparm (Linux), and I have noticed the volume group on the external PERC controller (RAID6 8x450gb 15k SAS) gets me an average of ~110MB/s Seq Read, and ~65MB/s Seq write in Windows and a buffered disk read of ~180MB/sec in Red Hat Enterprise 5 x64 (hdparm -tT /dev/sda).

Now, the internal PERC controller (RAID10 6x450gb 15k SAS) gets me a sluggish Seq read of ~50MB/s and a Seq write of ~30MB/s and as low as 5MB/sec. If I do a Storage vMotion to the External PERC, it gets a little better at ~95MB seq read and ~33MB seq write. I'm confused...all controllers have the same setup except the PERC 5/e in SAN2 (refer to my diagram for reference).

Any thoughts? I thought I had some consistent results, but a couple of the servers that were dropping is on the external array. Doesn't matter where I move them to. If I've confused you, let me know and I might be able to explain better.

Thanks again for all your help. I will update the IETd switchover later tonight...

Jason

[To-M]

Sorry for not getting back fast enough - very busy times .

Just let it run after the switch over - let the bond wait so we can trouble shoot one at a time.

Concerning the performance is was ready to jump on the disk drives but now in the end of your message that squelched that idea, unless firmware and or using 32bit mode made the difference of the drivers on the system - but don't think that is the case. We would have to see the logs to see if there is anything we can find to point us in the right direction. I would think the RAID 10 would do very well.

[jsinclair]

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

Todd -
I tried these settings again on my V6 beta SAN's, and once again all ESX servers (3.5 and 4) couldn't reconnect. I then tried to set them one at a time to see which setting caused the problem, and it's the MaxBurstLength setting. Every other setting allows the initiators to reconnect. Any other value I should try? Also, I have the logs from the balance-rr and you can download them below.

Thanks,
Jason

http://upload.vmind.com/web-pub/cera...6-18_21_16.zip

[To-M]

Engineers are investigating the bond issue as this is where they think the problem is also they are looking into the maxburst issue as well. I would leave the defaults for now. They will have to get some update for V6 to solve this issue with SCST.

[jsinclair]

Todd -
Sounds good. I will wait for advice then. It seems to be running OK now and not dropping machines, but the performance is still a little less to be desired.

If you need anything else from me let me know.

Thanks again for your help.

Jason