Open-E DSS performance

[tomtom901]

Hi All,

Currently I am running an Open-E system in testing mode as storage for the XenServer nodes. The current performance of the system bugs me a little, I am very curious to understand what are causing these non-optimal performance results. Or maybe the results are fine, I do not know and would like a second opinion.

The storage server setup
Dell R510
8 GB DDR3 1333 MHZ RAM
12x 300 GB SAS in RAID 10
PERC H700 controller, 1 gb cache, write back enabled
4x Intel PRO 1000 MBIT NIC

Network
HP Procurve 2510 as storage switch
Storage server is connected with 3x 1 gbit in LACP (802.3ad) trunk.
XenServer node is connected with 2x 1 gbit in LACP (802.3ad) trunk
Jumbo 9K frames enabled

From within a VM (4 cores / 4 GB RAM) in XenServer, I run the following dd command to execute a low-level data copy:

dd if=/dev/zero of=/var/tmpMnt bs=1024 count=12000000

Which returns:

12000000+0 records in
12000000+0 records out
12288000000 bytes (12 GB) copied, 109.673 seconds, 112 MB/s

I suppose some of the speed can be cache on the raid controller. However, when looking on the switch via SNMP, the traffic that flows during the copy action.. 400 mbit (see below). It isn't even close to the 2 gbit that (looking from the nodes perspective) I should have available.



Is this a lack in performance, should I be able to achieve more? Is this a configuration fault?

Any tips / advise would be appreciated.

[tomtom901]

Update with VMWare:

The following benchmarks have been achieved with VMWare (preferable hypervisor). I still have some questions concerning the seeks/s that have been achieved. Current (network) setup is 1x 1 GB from storage to switch and 1x 1GB from host to switch.

Benchmark specs:

[root@localhost ~]#  dd if=/dev/zero of=/var/tmpMnt bs=1024 count=12000000
12000000+0 records in
12000000+0 records out
12288000000 bytes (12 GB) copied, 100.451 seconds, 122 MB/s

[root@localhost ~]# ./seeker /dev/sda
Seeker v2.0, 2007-01-15, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [51200MB], wait 30 seconds..............................
Results: 386 seeks/second, 2.59 ms random access time

 
[root@localhost ~]# hdparm -Tt /dev/sda
/dev/sda:
 Timing cached reads:   22744 MB in  2.00 seconds = 11395.36 MB/sec
 Timing buffered disk reads:  276 MB in  3.01 seconds =  91.78 MB/sec

[Gr-R]

in your original post, have you tried to use MPIO instead of the bond?

in your second post, what are the details of the RAID.... controller, raid type, drive specs, etc...

[tomtom901]

in your original post, have you tried to use MPIO instead of the bond?

in your second post, what are the details of the RAID.... controller, raid type, drive specs, etc...

I have not tried MPIO, the article on MPIO is for vSphere 4, needs some customizations for 5, I am currently gathering them and getting them ready to mail to Open-E for an article on 5.

In the second post the same setup (hardware based) is being used so:

12x 300 GB SAS (ST3300657SS) in RAID 10
PERC H700 controller, 1 gb cache, write back enabled

[Gr-R]

let us know when you have the MPIO info. my experience tells me MPIO can typically be better then any bond.

regarding the hardware:
you posted:
Results: 386 seeks/second, 2.59 ms random access time

those drives can do :
Average latency2.0ms
Random read seek time3.4ms
Random write seek time3.9ms

so not bad really.

The PERCs dont get updated firmware from LSI very often, i suspect this can be an issue.

[tomtom901]

Update, I managed to get down MPIO. Some new benchmarks, some of them I question (mostly seeker). Besides from the guide, which I got down on vSphere 5, I made all the vSwitches eat Jumbo (9K frames).

dd command (2x executed with 2 hour gaps:

[root@localhost ~]# dd if=/dev/zero of=/var/tmpMnt bs=1024 count=12000000
12000000+0 records in
12000000+0 records out
12288000000 bytes (12 GB) copied, 74.6241 seconds, 165 MB/s

[root@localhost ~]# dd if=/dev/zero of=/var/tmpMnt bs=1024 count=12000000
12000000+0 records in
12000000+0 records out
12288000000 bytes (12 GB) copied, 73.7074 seconds, 167 MB/s

seeker:

[root@localhost ~]# ./seeker /dev/sda
Seeker v2.0, 2007-01-15, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [51200MB], wait 30 seconds..............................
Results: 912 seeks/second, 1.10 ms random access time

[root@localhost ~]# ./seeker /dev/sda
Seeker v2.0, 2007-01-15, http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [51200MB], wait 30 seconds..............................
Results: 934 seeks/second, 1.07 ms random access time

hdparm:

 Timing buffered disk reads:  302 MB in  3.01 seconds = 100.50 MB/sec

I think there still could be some as mentioned here and here.

Any thoughts?

[Gr-R]

MPIO.... how many paths? over 1GB?

[webguyz]

I believe the performance issue is on the Xenserver side. I have a DSS V6 and sharing it with a XEN network and Xenserver 6 network.

All servers using MPIO (2 1G cards) and the SAN has a 10G card. (using Block i/o)

using your test:

dd if=/dev/zero of=/var/tmpMnt bs=1024 count=12000000

On a linux server under XEN I was getting about 85MB/s (we have SATA drives in DSS) and on a linux server under Xenserver 6.0 I was only getting about 28MB/s

All settings on DSS had been optimized as per various posts in this forum for each volume. See various posts about ISCSI performance when goggling 'iscsi performance xenserver'

[webguyz]

Oops!

I have to take this back. I didn't realize I had my DSS Management IP added to the multipath of my 2 ISCSI nics, so that multipath -ll command showed 3 total active connections when when there should have been 2. The traffic was going out over my DSS mangement nic which has an MTU of 1500 instead of the 2 NICS that had Jumbo Frames enabled and an MTU=9000.

Fixed it and ran the test and getting the same throughput for the XEN servers as well as the Xenserver servers, about 85MB/s.