vSphere 4 - Support on the Way?

Printable View

Show 40 post(s) from this thread on one page

06-04-2009, 07:05 PM
1parkplace

vSphere 4 - Support on the Way?

Hi there,

Anyone have any idea as to a timeline when we might expect to see some compatability with VMWare vSphere4?

I recently upgraded ESX 3.5 to ESX 4 to test its compatability with DSS and it seems that they re-wrote their iSCSI stack code and there is some funky stuff going on.

I get dropped connections to LUNs and many path failures and reservation conflicts on the 4.0 server. When it occurs and the LUN mapping is dropped, the ESX 3.5 loses connection to it as well.

I can help the cause by providing Open-E access to my production ESX4 and DSS servers, just let me know.

1parkplace
06-04-2009, 07:46 PM
To-M

Thanks for providing us the use of your systems; we may need to back into that later. I did look at your logs and would like to verify some things concerning your Bond. Looks like Eth 2 and 3 have different speeds sets. This can cause issues and are these NIC's the same in terms of chips sets? We noted in the release notes that when using bonding to run a stress test with different chipset as we have seen issues also the speed difference can do this as well. Check the switch or force the speed in the Console from the Console Tools menu in the Modify driver settings. Then test again. Also you might want to change the iSCSI Daemon settings from the link below as I see PDU issues from the dmesg logs. More information about PDU can be found by Google.

This happens for LUN 1,2 and 3
iscsi_trgt: data_out_start(1037) unable to find scsi task 4f1b11f 8a93
iscsi_trgt: cmnd_skip_pdu(454) 4f1b11f 1e 0 4096

Try to set the in the console the following:
Ctl-Alt-W
Select Tuning Options
Select iSCSI Daemon options
Select Target Options
Select a target
Set MaxRecvDataSegmentLength and MaxXmitDataSegmentLength to 65536.

or try these settings:

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes
06-04-2009, 07:59 PM
1parkplace

hey todd,

The bond your seeing is strictly for the WebGUI for the DSS.

The iSCSI connection are only occuring on the eth1 which is the HP NC510C 10GBe NIC.

Should I set the following setting still or is this a compund issue?

Thanks!

Drew
06-04-2009, 08:08 PM
To-M

If you dont mind, let's kill the bond for now. Also make the changes for the iSCSI daemon settings as well.

I want to make sure there is nothing in our way in researching this issue. If there is anything else you can tell please do to help us.
06-04-2009, 11:34 PM
1parkplace

Response from someone in the VMWare community regarding my uploaded VMKernel Logs:
__________________________________________________ _______
There seems to be an issue with storage:

Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.023 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba33:CH:0 T:1 L:2 : Abort task response indicates task with itt=0x1107006 has been completed on the target but the task response has not arrived ...
Jun 3 01:01:18 vmhost-1 vmkernel: 3:15:42:27.272 cpu4:4239)WARNING: iscsi_vmk: iscsivmk_ConnSetupScsiResp: vmhba33:CH:0 T:1 CN:0: Task not found: itt 17854470

17854470(dec)=110700(hex)

1- There is an IO timeout (i.e storage is not responding to IO on time) which cases ESX iSCSI initiator to send an abort for that IO.

2- It appears that the storage responds to that with "task does not exist" but later the storage sends response the IO task. That is in violation of iSCSI protocol and ESX initiator drops the connection. This seems to keep happening very often.

ESX 3.5 s/w iscsi initator would just ignore that case but ESX 4 initiaror is very strict about protocol violation.

It appears you are using Open-E DSS, I do not think it is certified with ESX4 yet. Could you post the version of DSS you are using ?.

__________________________________________________ ____
06-04-2009, 11:47 PM
To-M

Thanks for the looking into this - I have forwarded this to engineers to verify. We are watching this thread. Also send me new logs once the changes have been made from the tuning options.
06-05-2009, 12:59 AM
1parkplace

Tuning Configuration is complete.

New log packages sent to your email.

Thanks!

Drew
06-05-2009, 06:12 PM
cschiff

vSphere compatibilty needed ASAP

Drew,

We too are facing a similar scenario; however we have not upgraded our ESX environment to vSphere 4.0 as I noticed they were not on the HCL for vSphere 4.0 Storage. However I really want to be able to upgrade to vSphere 4.0 so I am hoping they can resolve your issue and get recertified. However, I was not to optimistic on the speed of this happening as I was told by support that they may look at testing and recertifying sometime later this year, but as of yet no testing had been done with vSphere 4.0. I am going to continue to monitor your thread for progress resolving this issue. Hopefully since you already on vSphere 4 and need DSS to be compatible it doesn’t take until the end of the year to get that.
06-05-2009, 06:58 PM
cschiff

Drew,

I am also curious what version of DSS you are running?

Were you experiencing any iSCSI timeout errors when using ESX 3.5?

What RAID level are you running on your DSS SAN?

We had fought iSCSI timeout issues with DSS for many months. I tried configuring the DSS SAN with all of the recommended perfomance settings as specified by Open-e support. Things got a little better, but when the SAN was under moderate to heavy load we would still get CMD Abort and Task Not Found errors. What I found to be our ultimate solution was to rebuild our RAID set as RAID 10. Previously I was using RAID 5 and had my VMs split accross 2 DSS SANs, so each was had 50% of the ESX load. Now with the SANs at RAID 10, I am able to run ALL of my VMs on 1 SAN with better performance and NO iSCSI timeouts or errors, under any load.
06-05-2009, 07:18 PM
1parkplace

I'm running the 3513 build of DSS. I have two servers which I run DSS on, each server has 3 RAID configs and I map their LUNs to a single target. I have, on each server, different RAID setups ranging from RAID1, RAID5, RAID 10.

I also run these targets through a 10GbE Network, so maybe this is why I never ran into timeouts, are you using a similar setup?

On ESX3.5u3 I was not getting this same issue... however, I was experiencing finicky connection drops due to the use of the NetXen 10GbE NICs in my systems.

This weekend I will be installing Intel 2xCX4 10GbE NICs in my machines to alleviate that problem.
06-05-2009, 07:27 PM
1parkplace

my VMKernel logs have been fairly clean since I changed my iSCSI Daemon tuning options to what Todd recommended.

I am still getting these messages though in my VMKernel logs on ESX4.0:

Jun 5 11:08:57 vmhost-1 vmkernel: 6:01:50:02.470 cpu6:5337)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.469 cpu4:5340)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:13:57 vmhost-1 vmkernel: 6:01:55:02.470 cpu4:5340)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.469 cpu6:4171)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:18:57 vmhost-1 vmkernel: 6:02:00:02.470 cpu6:4171)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 839: Path 'vmhba33:C0:T1:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 842: Path 'vmhba33:C0:T1:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.701 cpu7:5341)ScsiScan: 105: Path 'vmhba33:C0:T1:L0': Peripheral qualifier 0x3 not supported
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 839: Path 'vmhba33:C0:T2:L0': Vendor: 'iSCSI ' Model: 'DISK ' Rev: '0 '
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 842: Path 'vmhba33:C0:T2:L0': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
Jun 5 11:23:57 vmhost-1 vmkernel: 6:02:05:02.702 cpu7:5341)ScsiScan: 105: Path 'vmhba33:C0:T2:L0': Peripheral qualifier 0x3 not supported
06-05-2009, 07:27 PM
cschiff

We are running DSS Build 3511 with ESX 3.5 U4. I have 2 DSS SANs each configured the same with 1 RAID 10 volume. Each SAN has 6 x 1 Gbps NIC ports. 2 are Bonded using Round-Robin for the GUI and 4 are Bonded using Round-Robin for iSCSI. These SANs only have 1 iSCSI target with multiple LUNs, each exposed only to a 3 ESX host Cluster.
06-05-2009, 07:36 PM
cschiff

Drew,

I found an interesting VMware forum on you latest ESX logs.

http://communities.vmware.com/thread/212783

I would read this because even though things seem ok and the errors have stopped. I have as run into the issue when we first got our DSS SAN and they had not received VMware Certification for 3.5 either, where I had complete LUNs disappear or just the data within the LUNs disappears. Be very cautious if there is any critical data on these SANs. I know I got pretty freaked out when a 1/4 of my VMs disappeared into thin air.

Just curious, do any of you LUNs have an ID of 0?
06-05-2009, 07:49 PM
1parkplace

Yea, I had actually read over that post early yesterday.

I had the same problem where my LUNs would disappear in a sense that when I browsed the Datastore on ESX all data was gone.

I had to reboot the ESX hosts and the DSS about 2 times before anything showed back up.

I later found that logging into the DSS and managing the target, that unmapping and remapping the LUN quickly fixes the problem and all of the VMs show back up and continue operating properly.
06-05-2009, 08:22 PM
To-M

cschiff - I sent you an email on your forum account on obtaining our new DSS product version. I am only sending this to all VMware users now to test the new version as we have a new target sw package. Engineers cannot test every environment and we are hoping for some feedback and see if this fixes so many issues.

1parkplace has already received it.

Any one else willing to provide feedback before we release please ask me but provide feedback - please!
06-05-2009, 09:24 PM
cschiff

Todd,

I never received your email???

We do have a 2nd SAN that I would be willing to load with the test copy of DSS. Once I have upgraded the backup SAN, I will upgrade 1 of my ESX server to vSphere 4.0 and provide you with detail positive and negative on how my testing is going.

Thx,

Christian
06-05-2009, 09:55 PM
To-M

Check your forum email account - just sent it again. If you dont get i will send using your other email address.
06-05-2009, 10:17 PM
jsinclair

vSphere4

I am willing to test out the new release. In fact, I have just started testing with 2 - DSS servers, iSCSI failover, and Volume Sync with ESX 3.5. I'm glad I saw this post, as we are planning on upgrading our VMware enviornment to vSphere 4 in the next week. Please let me know if I can participate in the Beta.

Thanks,
Jason

Info: 2x Dell PE 2950 with Perc 6/i and MD1000 split on 2x Perc 6/e
(4TB total with 15k SAS drives)
06-05-2009, 10:20 PM
To-M

Many thanks Jason!

Please provide any comments that would help our engineers. Sometimes the customers see things we cannot.

Check your forum email.
06-05-2009, 10:41 PM
cschiff

Todd,

I got the fille. I will update my backup SAN tonight and upgrade one of our ESX servers on Monday. I should hopefully have some feedback for you later on Monday or Tuesday at the latest.

Thx,

Christian
06-05-2009, 10:43 PM
To-M

Most appreciated!
06-06-2009, 12:24 AM
1parkplace

Set up my test environment about 10 min ago.

Installed DSS v6 Beta and vSphere 4 on some junk machines today and setup the environment. Currently installing the Win2k3 VM and I will bang at it for a day or so and send Todd the logs from both systems to see if there is anything weird with the interaction of the machines.

Drew
06-06-2009, 05:11 PM
fobe

Is V6 beta already available? I can't find any links on te site...:(
06-08-2009, 01:00 AM
jsinclair

Todd -
Quick question(s).

Currently all our production VMWare 3.5 is on an OpenFiler iSCSI SAN. We are moving forward with Open-E because of the iSCSI fail over and volume replication. I started to setup 2x Open-E trials and had the volume replication and iSCSI failover working. My plan was to move all the VM's w/ Storage VMotion from openfiler to open-e once I was confident the performance and stability was what we were looking for and then purchase 2 - 4TB DSS licenses. Once this was complete, we would upgrade our VMWare 3.5 to vSphere 4. After reading this post, I am now hesitant to pump the brakes on this approach. I did get the beta from you (thanks very much) and have the second SAN upgraded. I am rather impressed so far and actually excited for this release because you have just fixed my USB issues with my Dell 2950's. I have a 4GB SATA DOM that I was able to install version 6 on with no issues! This was how OpenFiler was setup so I happen to have extra SATA DOMs.

Now...since I am testing these systems on our production vmware, I want to make sure I don't get stuck in a bind before upgrading VMware. Since v5 of Open-E will not work with vSphere 4, but testing goes well with the beta of v6 what are the next steps? Will version 6 be available within the 60 day trial? If I go ahead and purchase the 2 version 5 modules, will I be able to use version 6 beta with vsphere 4 without it expiring and losing data?

I hope this makes some sense and if not, please feel free to send me a private message and we can arrange a call. I want to help out in the beta testing, but I also have a deadline I must meet with providing redundant storage for vmware and upgrading to vsphere. I might be able to dig up some test hardware, but I don't think I could really put it to the test like I could with our production :-)

Any questions, let me know...

Jason
06-08-2009, 06:22 AM
1parkplace

I'm essentially in the same boat.

My test setup with the v6 has proven to not cause errors on vSphere4, at least in the VMKernel log aspect. I have yet to see the logs from DSS v6, I will be sending my log bundles to Todd tomorrow morning.
06-08-2009, 04:34 PM
To-M

Jason,

You can use the beta version and if all goes well with your testing you can purchase the DSS V6 and restore your settings that you saved from the beta version from the Maint. > Misc. GUI section - please take all options.
06-08-2009, 04:55 PM
jsinclair

Todd -
Sounds great. Will V6 be available before the 60 day trial ends? Can I purchase a key direct, or through a reseller?

Thanks again!
06-08-2009, 05:08 PM
To-M

DSS V6 will be available before the 60 day trial ends and you will need to purchase thru a reseller of distributor for your location from link below.

http://www.open-e.com/partners/solution-providers/

or

http://www.open-e.com/where-to-buy/distributors/
06-08-2009, 07:27 PM
cschiff

Todd,

If we currently have version 5, do we have to purchase version 6 or will this be a free upgrade?

Thx,

Christian
06-08-2009, 07:57 PM
To-M

I know that there will be an upgrade cost but not sure how this will be as I don't handle the pricing. Pricing will be announced later.
06-08-2009, 08:33 PM
gregg_hughes

What's in v6.0?

Is there any word or doc on what's in v6.x? What changes from the v5.x?
I'd like to see changes so we can prepare and decide.

Gregg
06-08-2009, 08:38 PM
To-M

We will have the list ready in July as we are still working on many other issues. Mainly new iSCSI target (SCST), Persistent Reservations for Win. 2008, NDMP ver. 4 and new GUI.
06-10-2009, 04:43 PM
jsinclair

I have upgraded to vSphere 4 and I am currently running with 4 x ESXi 4 servers and 2 x Dell 2950 w/ DSS v 6. The volumes are sync replicated with iscsi failover. There have been no issues so far, but I haven't migrated all the machines (just a few) over to the DSS SAN. Once everything is migrated, I will let it run and send the logs.

Todd - Are you anticipating a July release of v. 6? I just want to ask (one more time) before I move production data on it :-)

Also - as far as my iscsi config, should I use SCST instead of IETD? If the volumes are setup with File I/O and initialized, should I also configure in the console, tuning, to use File I/O with iscsi? Just wanted to make sure...

Thanks again!

Jason
06-10-2009, 04:51 PM
To-M

Jason,

Thanks for testing this for our engineers as we have not had the time and are pushing the final release for the first or second week of July - without fail!

I would use the SCST (Default in DSS V6) and with File IO. If you create them in begging you should not have to switch them and if possible do the initialization in the begging as well.
Though you can switch in the tuning console, I like to get it all done before you do replications and other function.
06-10-2009, 06:30 PM
Robotbeat

SCST is pretty sweet. I'm on the SCST mailing list, and there are new features all the time, with excellent performance on the basic iSCSI stuff.

A nice thing about SCST is that it supports exporting devices using:

iSCSI, of course

Fibre Channel (even support for 8Gbps)

SCSI RDMA Protocol (SRP), a very low latency iSCSI-like connection using Infiniband (which has 40Gbps over a single connection), which is an inexpensive interconnect that almost all the fastest supercomputers use

Fibre Channel over Ethernet (FCoE)

Parallel SCSI (!)

SAS (even the new 6Gbps cards)

Also, you can use file-io, block-io, or even just pass-through (for sharing old Magneto-Optical or Tape systems over your network of choice).

Open-E doesn't support all of these interfaces, yet, but if enough people would use a given feature, I'm sure they'll get around to implementing it.
06-10-2009, 07:24 PM
smasher953

Todd -- I also have some vsphere 4 systems that I'd like to add onto my DSS system. Is the v6 beta still available for me to do some testing with?
06-10-2009, 07:29 PM
To-M

No problem - just provide everyone with your thoughts and any performance information would be good. So check you forum inbox email.

Anyone has Hyper-V that they can test?

Thanks guys!
06-11-2009, 04:03 AM
jsinclair

I have roughly 40 virtual machines that are (currently) storage v-motioning to a DSS v 6 cluster in vSphere 4. So far no issues. I am using the SCST and file i/o with jumbo frames enabled on the sync interface (crossover cable). They will continue to migrate overnight (gotta love task scheduling). I will update everyone with my experiences.

Just a quick note...the virtual machines are being migrated from an Openfiler system. The connection was lost for some reason to the OF system and I had to reload the iscsi service to bring back my VM's on OF. The VM's already migrated to Open-E were the only ones that didn't drop :-) So far so good...

More to come...
06-11-2009, 05:17 AM
To-M

Jason, Thanks for the update :) - Engineers are looking at this post.
06-16-2009, 06:27 PM
jsinclair

Beta v6/vSphere Sync Replication - Slow Performance

Todd -
I have been experiencing some strange performance problems in the v6 beta cluster/failover setup I have. They are both Dell PE2950's with 2 PERC controlelrs and an MD1000 attached.

I am including a link to download the logs for both servers as well as a visio drawing of what I have setup (I could type it all out, but I work better with pictures :) ). What I am seeing is very sluggish virtual machines at no specific time or reason during the day. Currently I have 4 x ESXi 4 (vSphere 4) which are Dell 1950's with 4 NICS (2 for iscsi vmotion, and 2 for data). Each ESX server has roughly 6-7 virtual machines running on it. There is nothing very disk intensive at this time, just a Blackberry Enterprise server, Exchange front end, Windows update server, and other misc Windows 2003 and Linux servers and all SQL-based applications connect to a dedicated physical SQL 2005 server.

If you could take a peek at the logs and let me know if you see anything, that would be great. Maybe you can spot something I can't. The ESX servers are not dropping the iSCSI connections, the virtual machines just run very slow at times and sometimes temporarily freeze for a couple seconds.

If there is any other info you need, please let me know.

http://upload.vmind.com/web-pub/cera...etav6-Logs.zip
http://upload.vmind.com/web-pub/cera...SS-Storage.png

Thanks for your help!

Jason

JSinclair@vmind.com

Show 40 post(s) from this thread on one page