DSS / iSCSI Performance

Printable View

03-27-2007, 01:23 PM
markstaller

DSS / iSCSI Performance

I'm currently with designing the future of our storage infrastructure where DSS will be an important building-block.
Now I'd be very interested in sharing some performance numbers for comparison, to see what others get and which components have a large impact on performance.
For now I think I'm still far away from whats possible in terms of performance.

In the following I will post some results with my current setup, I'd really appreciate some other results or tips on how to measure and what components have a big impact.
I guess it would help much if Open-E would publish some performance-results from various setups, so there's something to look at when deciding what to buy..
The usual recommendations, mostly by storage vendors, aren't very helpful (like use FC, use 15k, use only SCSI, use RAID10 - this surely helps but also raises cost and if one needs 100MB/s why should the HW be able to do 500).

Machines:
DSS "e4050":
- Dual Xeon 2,66G, HT, 1GB RAM, Intel SE7501CW2 Board
- 3ware 9500S-8, BBU, Bios FE9X 2.08.00.005, Driver 2.26.02.008
- 8xBarracuda ST3400832AS as RAID-5 with 1 Hot-Spare
- OS DSS 1.15
- NIC: 2x Intel Pro1000 MT Dualport Server, e1000 PCI:66MHz:64-bit with TOE, Driver 7.3.20, No bonding

Host "e4051":
- Dual Xeon 3GHz HT, 6GB RAM, Intel SR1435 Board
- 3ware 9550SXU-4L, Firmware FE9X 3.04.01.011, BIOS BE9X 3.04.00.002, Driver v2.26.02.004, BBU, RAID-1 with 2x ST3400832AS (only for booting, not relevant for these tests)
- OS Ubuntu 6.06LTS, 2.6.15-28-server
- open-iscsid version 0.3 connected to DSS using defaults. No CHAP, no IPSec
- NICs: eth0 sk98lin, 8.23.1.3, TOE on; eth1: e1000 7.0.33-k2, TOE on

Now I did some tests with bonnie++ (other tools better?) and get IMHO far too slow results :
Server Size SeqOut-Chr K SeqOut-Block K SeqOut-ReWrite K SeqIn-Chr K SeqIn-Block K Random seek/sec Seq Create/sec
e4051-eth0 10G 35.698 64.390 21.598 22.855 42.267 196 2.783
e4051-eth1 10G 34.898 48.866 19.439 20.926 39.717 195 2.746

I did the same on the DSS-host (before installing DSS) and got:
e4050 2G 34.021 76.578 31.682 28.646 176.187 418 2.639

- So it's unlikely the local array on the DSS is the bottleneck
- Write-performance via iSCSI seems quite well, this might be really close to the max of the RAID-5 in the DSS
- Read-performance over iSCSI is with 42MB much too slow
- Tried with both NIC's, but reading is slow in both cases
- How to verify TOE is enabled on the DSS?

No further tuning was done, so pretty much all on default-values on both sides. I guess the main thing is networking-performance as I get also odd results from iperf between servers (but I cannot do iperf against the DSS).
I manily wonder wether it's tuning some parameters, using Jumboframes (which I got worse results with testing a year ago), using HW iSCSI HBA's (expensive), using NIC-bonding or is it just about fiddling some iSCSI or TCP-parameters..

Michael
03-27-2007, 03:33 PM
To-M

This is good stuff and I agree we should be posting more speed and setting results. The only problem is we deal with so many resellers and customers and the results are never the same. This is due to the fact that when you introduce supporting so many hardware components as we do (others do not and this is where we excel) to provide a wide range of product support with our software you now usher in different results from all the different components that integrate with each other. Not to mention there is no way anyone can test all the variables in a business network environment (as no 2 networks are usually the same - use of R/W, I/O...). This is why I like what you are doing here as others will come in and say " I am using a Top End Supermicro Motherboard, fastest RAM, Areca 1280 (Memory added to the Max. and striped to the highest value), bonding with 802.3ad (with top end managed performance switches) and the host systems with the latest most recent hardware components.... You get the idea. I also wanted to let you know that the DSS requires CPU 3.0GHz or faster (higher cache the better) yours is 2.66GHz. Any way you can test this with a higher CPU as recommended and newer motherboard.
As with any new products and software you always want to provide latest and fastest as the product is designed for this. Also try to set your RAID to highest chunk size 256+.
Check the CPU, MEM and NIC stats under Status - Hardware - Function: Server statistics to see what’s happening.

By the way I had many engineers call me and tell me their is a firmware update from
3Ware to fix some performance issues - this is to be released in Q2 of 07.

Another note, allot of engineers had reported that the Areca 1280 ROCKS on performance even on a RAID 6 is only 5% less then the fastest 3Ware RAID5!!!

I also did some research concerning the 3Ware and they have some speeds with charts for RAID5 and others with Databases and Webservers.

http://www.xbitlabs.com/articles/sto...-9500s8_8.html

I thought this was interesting concerning the editor’s comment on 9500.

"The influence of WT-caching on the RAID10 array performance is negative in all work modes with available write requests. The maximum performance drop was 353%. However, as we have already mentioned above, disabling the controller cache leads to a performance drop in RandomRead mode when there are no write requests in the queue, which is bad. So, we can conclude that enabled WT-caching reduces the performance significantly. Therefore, it is really hard to say what caching type is preferable for most efficient work. The reliability is higher in WriteThrough mode, however, in WriteBack mode the performance boost is definitely more tempting"

You can experiment with NIC parameters by accessing the Consol tools then ctrl. + alt + t
then select Modify Driver Options. Also try to test with the iSCSI daemon options ctrl-alt-w --> tuning --> iSCSI deamon options. Here is a great link to understand in details some of the terms and settings- http://www.zvon.org/tmRFC/RFC3720/Output/chapter12.html

Also keep in mind that the reason other vendors don't like to report to much on speeds is the variables involved - unless you pay for their proprietary hardware and software $$. We are a fraction of the cost of standard equipment sets like EqualLogic (starting in the $40k+++) which by the way does not offer you the choice that we supply for all the different compatibility listings for DSS - which is extremely inexpensive.

Again I would like to thank Michael for his efforts in starting this as this can help everyone
to benefit as we all love that SPEED. Some of the comments are for everyone to keep in mind. But going forward we need this information!! Let’s post more settings and speed results - I like it!

Thanks again Michael!!
03-27-2007, 05:47 PM
markstaller

Quote:

Originally Posted by To-M

This is good stuff and I agree we should be posting more speed and setting results. The only problem is we deal with so many resellers and customers and the results are never the same.

It's quite clear that these results differ and aren't comparable, but I guess it stills gives some insights on what can be expected for a certain combination and it's good at least for building some rules of thumb.

Quote:

Originally Posted by To-M

I also wanted to let you know that the DSS requires CPU 3.0GHz or faster (higher cache the better) yours is 2.66GHz.

Really, I didn't thought that a CPU above some P4 on a DSS would have a significant impact on iSCSI-I/O (significant compared to Disks, Raid). You mean the bigger 2nd-Level cache on the CPU, not RAM right?
Anyway, also for this some benchmark for comparison would help to decide for anybody what's worth it..

Quote:

Originally Posted by To-M

As with any new products and software you always want to provide latest and fastest as the product is designed for this.

Sure, this box is ~1y old (this is also not the box that will have high IO in future) but it's my test-environment for now..

Quote:

Originally Posted by To-M

Also try to set your RAID to highest chunk size 256+.

-> just changed the stripe-size from 64k to 256k; will test afterwards.

Quote:

Originally Posted by To-M

Check the CPU, MEM and NIC stats under Status - Hardware - Function: Server statistics to see what’s happening.

This returns an empty page on my DSS ? I just didn't had the time to complain so far :D

Quote:

Originally Posted by To-M

By the way I had many engineers call me and tell me their is a firmware update from
3Ware to fix some performance issues - this is to be released in Q2 of 07.
Another note, allot of engineers had reported that the Areca 1280 ROCKS on performance even on a RAID 6 is only 5% less then the fastest 3Ware RAID5!!!

I also did some research concerning the 3Ware and they have some speeds with charts for RAID5 and others with Databases and Webservers.

http://www.xbitlabs.com/articles/sto...-9500s8_8.html

Although I guess I'm not after a controller problem when suffering from 40MB/s read in a 7-disk RAID5 while writing 60+, it's interesting. It simply tells me to buy another controller that isn't broken under this or that circumstance for nearly the same price..

Regarding NIC and iSCSI-Parameters honestly I'd prefer to rely on whatever your engineers found to be the best (which isn't true for the initiator side for sure..). All of such fine-tuning is extremely time-consuming and leads to complex settings and mostly unreproducible results when adding i.e. another initiator.. I'm happy for now when reaching 80% of whats possible with 20% effort - if you know..

Quote:

Originally Posted by To-M

Also keep in mind that the reason other vendors don't like to report to much on speeds is the variables involved - unless you pay for their proprietary hardware and software $$. We are a fraction of the cost of standard equipment sets like EqualLogic (starting in the $40k+++) which by the way does not offer you the choice that we supply for all the different compatibility listings for DSS - which is extremely inexpensive.

That's why I'm with DSS :D I can solve any performance issue with buying a 400% oversized system always, but the glue for my "just in time" datacenter is to have what I need now and nothing more, that saves thousands each year.
Also on another note, not all of these "enterprise" systems are that drop fast.. I did perf-test with several E**, N***** boxes before and most of them underperformed by ages as well (but for sure for another 30k$ you could buy FC-disks and x more controller to compensate that..)

I'll try to get some more results but resources are - as always - limited.. For the next two DSS planned I'd wanted to have a better understanding where to invest; not buying the most expensive from everything, which also helps but costs much more money than focusing on the relevant parts.

Michael
03-27-2007, 07:01 PM
To-M

This returns an empty page on my DSS ?
Try several times to hit the Apply or Remove button to gain access. New version has resolved this in Server statistics, the new DSS version is in finial engineering phase and will be released next week. Leaving the defaults for the iSCSI-Parameters are as our engineers have recommended and as for the NIC there are just too many NIC's along with updated firmware's to compare with the fine tuning. This is where we focus on providing the support for as many hardware components as possible instead of just focusing on couple (as do with some vendors). Too many customers out way this by 95% asking for a larger selection. I have found some engineers having Intel systems have reported that the Intel I/O AT settings performed well - this is located in ctrl. + alt. + t then select Boot options >System Architecture.
03-27-2007, 09:54 PM
markstaller

Server stats work now after hitting remove several times as you wrote.. It's somehow *dead slow* but doesn't matter as it's a subset of what I already monitor by SNMP remotely ;)

Michael
03-27-2007, 10:19 PM
To-M

This should respond quickly. Any chance to see what the CPU load is or was during the time when you tried to access it. Also was the rest of the GUI slow as well?
03-29-2007, 12:55 PM
markstaller

I'm waiting for my RAID-5 to be migrated from 64 to 256k as recommended, now in the meatime I added channel-bonding (802-3ad):
- Two-port Intel Pro1000 MT Server Adapter
- Cisco Cat3750G config on both If to DSS: " channel-group 2 mode active"

Code:

s04602#sh lacp int Flags: S - Device is requesting Slow LACPDUs F - Device is requesting Fast LACPDUs A - Device is in Active mode P - Device is in Passive mode Channel group 2 LACP port Admin Oper Port Port Port Flags State Priority Key Key Number State Gi1/0/18 SA bndl 32768 0x2 0x2 0x12 0x3D Gi1/0/19 SA bndl 32768 0x2 0x2 0x13 0x3D

So it seem to run LACP..

But Traffic is going only over the second Interface (eth1 & Gig1/0/19) from the DSS, at least for one connection.. I either misunderstood lacp or something is wrong. At least this bonding won't help anything to improve iSCSI-performance..

Michael