vSphere 4 - Support on the Way?

[jsinclair]

I have roughly 40 virtual machines that are (currently) storage v-motioning to a DSS v 6 cluster in vSphere 4. So far no issues. I am using the SCST and file i/o with jumbo frames enabled on the sync interface (crossover cable). They will continue to migrate overnight (gotta love task scheduling). I will update everyone with my experiences.

Just a quick note...the virtual machines are being migrated from an Openfiler system. The connection was lost for some reason to the OF system and I had to reload the iscsi service to bring back my VM's on OF. The VM's already migrated to Open-E were the only ones that didn't drop :-) So far so good...


More to come...

[To-M]

Jason, Thanks for the update - Engineers are looking at this post.

[jsinclair]

Todd -
I have been experiencing some strange performance problems in the v6 beta cluster/failover setup I have. They are both Dell PE2950's with 2 PERC controlelrs and an MD1000 attached.

I am including a link to download the logs for both servers as well as a visio drawing of what I have setup (I could type it all out, but I work better with pictures ). What I am seeing is very sluggish virtual machines at no specific time or reason during the day. Currently I have 4 x ESXi 4 (vSphere 4) which are Dell 1950's with 4 NICS (2 for iscsi vmotion, and 2 for data). Each ESX server has roughly 6-7 virtual machines running on it. There is nothing very disk intensive at this time, just a Blackberry Enterprise server, Exchange front end, Windows update server, and other misc Windows 2003 and Linux servers and all SQL-based applications connect to a dedicated physical SQL 2005 server.

If you could take a peek at the logs and let me know if you see anything, that would be great. Maybe you can spot something I can't. The ESX servers are not dropping the iSCSI connections, the virtual machines just run very slow at times and sometimes temporarily freeze for a couple seconds.

If there is any other info you need, please let me know.

http://upload.vmind.com/web-pub/cera...etav6-Logs.zip
http://upload.vmind.com/web-pub/cera...SS-Storage.png

Thanks for your help!

Jason

JSinclair@vmind.com

[To-M]

Jason - Nice details on the graphics!!

I have forward this onto the engineers that are involved with DSS V6.

Logs from what I saw did not show any issues with packet loss only some target settings that I may want you to test with and change on our end from the console in CTRL ALT W then Tuning Options then iSCSI daemon settings and then select the target and try these settings. It will do a reset on the VMware side.

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

[jsinclair]

Todd -
Thanks! I just find it easier to visualize something than write it down :-)

I tried the settings you suggested, and it lost my iscsi connections on all vmware ESX servers (2 x esxi 3.5, and 4 x vsphere 4). I was unable to get them restored, even after a reboot of an ESX server, vCenter server, and SAN1. I had to set everything back to the default settings to make everything work again.

Any idea what I may have done wrong?

Thanks again,
Jason

[To-M]

Engineers would like to see if you can test with the IETd instead of the default SCST - to change this go to the console then enter CTRL + ALT + W then Tuning options then iSCSI daemon then iSCSI solution then select IETd - you will lose connections from a reset and you will need to reconnect from the initiator side.

Also can you try without the bond as well. Then send the results with logs to support@open-e.com in the subject line of the email please copy and past this

Re: [Ticket#1009585] DSS V6 tests are slow with VMware ESX 3.5 and 4.0

We have a team watching this ticket.

[jsinclair]

Todd,
No problem. I can change to IETd, but since it is during business hours, I may have to wait. If I failover to san2, change san1 to IETd and then failback should that help prevent the downtime?

Also - since the bond is the main iscsi interface which has the virtual IP assigned to it, how will this affect my setup? Should I try a different bonding method?

Thanks,
Jason

[jsinclair]

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

Todd -
I tried these settings again on my V6 beta SAN's, and once again all ESX servers (3.5 and 4) couldn't reconnect. I then tried to set them one at a time to see which setting caused the problem, and it's the MaxBurstLength setting. Every other setting allows the initiators to reconnect. Any other value I should try? Also, I have the logs from the balance-rr and you can download them below.

Thanks,
Jason

http://upload.vmind.com/web-pub/cera...6-18_21_16.zip

[To-M]

Engineers are investigating the bond issue as this is where they think the problem is also they are looking into the maxburst issue as well. I would leave the defaults for now. They will have to get some update for V6 to solve this issue with SCST.

[jsinclair]

Todd -
Sounds good. I will wait for advice then. It seems to be running OK now and not dropping machines, but the performance is still a little less to be desired.

If you need anything else from me let me know.

Thanks again for your help.

Jason