I'm currently with designing the future of our storage infrastructure where DSS will be an important building-block.
Now I'd be very interested in sharing some performance numbers for comparison, to see what others get and which components have a large impact on performance.
For now I think I'm still far away from whats possible in terms of performance.
In the following I will post some results with my current setup, I'd really appreciate some other results or tips on how to measure and what components have a big impact.
I guess it would help much if Open-E would publish some performance-results from various setups, so there's something to look at when deciding what to buy..
The usual recommendations, mostly by storage vendors, aren't very helpful (like use FC, use 15k, use only SCSI, use RAID10 - this surely helps but also raises cost and if one needs 100MB/s why should the HW be able to do 500).
Machines:
DSS "e4050":
- Dual Xeon 2,66G, HT, 1GB RAM, Intel SE7501CW2 Board
- 3ware 9500S-8, BBU, Bios FE9X 2.08.00.005, Driver 2.26.02.008
- 8xBarracuda ST3400832AS as RAID-5 with 1 Hot-Spare
- OS DSS 1.15
- NIC: 2x Intel Pro1000 MT Dualport Server, e1000 PCI:66MHz:64-bit with TOE, Driver 7.3.20, No bonding
Host "e4051":
- Dual Xeon 3GHz HT, 6GB RAM, Intel SR1435 Board
- 3ware 9550SXU-4L, Firmware FE9X 3.04.01.011, BIOS BE9X 3.04.00.002, Driver v2.26.02.004, BBU, RAID-1 with 2x ST3400832AS (only for booting, not relevant for these tests)
- OS Ubuntu 6.06LTS, 2.6.15-28-server
- open-iscsid version 0.3 connected to DSS using defaults. No CHAP, no IPSec
- NICs: eth0 sk98lin, 8.23.1.3, TOE on; eth1: e1000 7.0.33-k2, TOE on
Now I did some tests with bonnie++ (other tools better?) and get IMHO far too slow results :
Server Size SeqOut-Chr K SeqOut-Block K SeqOut-ReWrite K SeqIn-Chr K SeqIn-Block K Random seek/sec Seq Create/sec
e4051-eth0 10G 35.698 64.390 21.598 22.855 42.267 196 2.783
e4051-eth1 10G 34.898 48.866 19.439 20.926 39.717 195 2.746
I did the same on the DSS-host (before installing DSS) and got:
e4050 2G 34.021 76.578 31.682 28.646 176.187 418 2.639
- So it's unlikely the local array on the DSS is the bottleneck
- Write-performance via iSCSI seems quite well, this might be really close to the max of the RAID-5 in the DSS
- Read-performance over iSCSI is with 42MB much too slow
- Tried with both NIC's, but reading is slow in both cases
- How to verify TOE is enabled on the DSS?
No further tuning was done, so pretty much all on default-values on both sides. I guess the main thing is networking-performance as I get also odd results from iperf between servers (but I cannot do iperf against the DSS).
I manily wonder wether it's tuning some parameters, using Jumboframes (which I got worse results with testing a year ago), using HW iSCSI HBA's (expensive), using NIC-bonding or is it just about fiddling some iSCSI or TCP-parameters..
Michael