DSS FC write problems

Hi,

I have been testing various block and frame size settings to see what performs the best and I have run into write performance issues.

The SetupDSS 3511 with a Qlogic QLE2462
XenServer 5 (Update 3) with a Qlogic QLE2462
Qlogic 5600 switch in between

I created a 10GB volume on DSS for each block size (512, 1024, 2048 and 4096). I attached each as a Storage Resource to XenServer.

One at a time I created a 5GB partition and attached them to a Win2k3 VM server. The only one Win2k3 could see properly was with the 512 block size. All the others did not show the proper size partition and would not initialize. I also changed the FC frame size settings on both ends (512, 1024, 2048), but it did not make a difference.

I then proceeded to perform READ tests. The best numbers were with a 512 frame size. Then I started WRITE testing and this is where my trouble began. I would get a quick burst, then the test would lockup.

The next thing I did was to do a simple file copy. I copied 2GB of data to the partition. The copy would go in short bursts and take forever to complete.

For comparison I attached the XenServer to a volume on our SANMelody SAN and did the same set of test. While the performance was less then what I was getting with DSS, the READ and WRITE operations worked perfectly.

So my conclusion, being that I am using the exact same software and hardware for testing, is there is something wrong on the DSS side.

One other note -- I found that if I create a FC volume on DSS and attach it to the XenServer, then detach it from the XenServer and change something on DSS, like delete and created another volume, I have to reboot both xen and DSS for xen to see the updated volumes correctly.

Any suggestions on how to fix the WRITE issue??
Mike

Are you using the "default" blocksize when you format the drives as ntfs? When I tested FC volumes with different blocksizes set on the DSS interface, I manually set the blocksize in windows as I was formatting them as ntfs. This was also with windows 2003. Have you done anything like that?

I did format with default. Just now talking with Todd, I reformated with 512 and 4096 and got the same bursty respone on writes.

HI hughesmikea

Your Qlogic 5600 switch settings are they set to auto?
If so set them manually.

I changed the switch ports for DSS from auto negotiate to 4GB. No difference. I/O guard is set to auto.

I did not change the ports for Xen, because I do not have this issues from Xen to SANmelody on the same switches.

thanks
mike

Hi hughesmikea

What RAID level are you using?
Can you try enabling cache and read ahead on your RAID controller.
how much RAM do you have ?
Is it locking up or is the write fast in the beggining and then slowing down?

Have you compared it to iscsi or nas performance on the same disks/dss?

gentlemen,

Server is a brand new PowerEdge 2970 with a PERC 5/i RAID controller and 4GB RAM
(6) 1TB SAS Drives in a RAID 5 with writeback cache enabled (default)

Just created an iSCSI volume and attached my PC to it. I copied a 2.5GB file across smoothly.

When copying acrosss FC, the copy has an initial burst, then pauses for a while, then bursts again, then repeats until complete.

mike

Forget to ask or maybe you did test this already but what about directly connecting w/o the switch. I know that the Datacore you didn't have to do this but just a thought.

Hi hughesmikea

can you manually set the I/O Gaurd

Good Morning,

OK, I went port to port from Xen to DSS and it worked fine. Then I wen back through the switch and it is NOW WORKING. No changes were made to the FC HBA's or switch.

Any thoughts on how this could be??

mike

Perhaps it was a problem with the auto-setup inside the switch?

Maybe set the ports to Point to Point first the other, not sure as I am light on the FC switch end as I dont have one.

Hi hughesmikea

Your I/O Gaurd is is that still set to Auto, if so please try manual setting.
Someone told me before that fibre channel switch settings must be manualy set.
and do not leave any set to auto.

Hi Guys,

OK, I have played with every setting under the sun and reset everything to factory defaults (not that much was changed to begin with). Right now I am at factory defaults on the switch and HBA's and everything is working perfectly.

At one point today, I shut down everything so I could take the FC card out of DSS and reflash it. When I brought it backup up, the write issue came back. At that point I started playing with all the settings to see if it would make a difference. Nothing did. I set everything back to defaults and the problem was still there. Then without shutting anything down, I reset, through the switch admin GUI, the port DSS is plugged into and the issue went away. And before you ask, I statically setting the speed to to 1, 2, and 4GB and it did not make a difference, nor did the i/o guard.

My conclusion is either a port syncing issues on the switch or a driver issue on DSS. From a lamen's perspective, I would sway towards driver, because we not have this issue with SANMelody and the same server. But like I said, I am a lamen when it comes to FC and drivers.

A couple of thing I found through all of this that would be nice to have resolved or added are:

1) Any time you make a change in the Fibre Channel config you have to reboot the server. This is definitely not good for a production environment.

2) When initializing a new volume the performance of all the other volumes comes almost to a halt. I initialized a 2TB volume today, which took quite some time, and the 15GB volume I was testing with became pretty much non-functional. Again, this is something that is not good for a production environment.

3) The Qlogic CLI (Ctl-Alt-F) is not there, so to make any changes to the FC HBA you have to shut down the server and do it through BIOS. Even then you cannot change all the possible settings and troubleshoot.

4) The ability to see what Initiators are connected to a volume (with statistics) would be really handy. Both FC and iSCSI

5) The ability to see the auto generated name of a volume. This is the one the initiator sees in its discovery of LUN's.

6) The ability to change the name of a logical volume or set it at the time of creation.

Thanks for you help. Let me know what you think.
Mike

Mike - Thanks for the update!

I also want to thank the guy's on the forum especially for contributing to Mikes needs.

I think Mike makes some very good points and we are aware of these issues already and wanted to let you that our engineers review the posts as well.

So if you can give us some time to let our FC guy work on making it better in our future releases then we will have this done with a better price point then our competitors :)

Mike - Do you know how much faster we where from the Datacore rates, would be good to know for the guys who helped out as well. I know Robotbeat would love to know :) as well.

Another thing i discovered about setting up some fc target is Machine Zoning,
I mean make a zone for each machine connected to the lun even if you share your volume through multiple machine. On Qlogic switch like sanbox2, do not enable I/o guard on your target machine.

My setup work like a charm with a metasan controller.