Very low performance issue

Hi,
I still have a huge performance issue with Open-E DSS v6.

My server configuration :
Xeon L5520 (4 quad cores)
8DDR3 ECC
5x Crucial RealSSD 256Gio (290Mio/s in SATA2 per SSD)
3x Intel e1000 Gigabits Ethernet

The 3 networks cards are in bonding (802.3ad) and the SSDs are in software Raid5 (using DSS) so I should get a 3Gbit/s connection and a very high bandwidth capacity for the disks (I except more than 800Mio/s in local).
But the problem is that I don't even managed to have more than 100Mio/s per iSCSI volume ! The speed is not stable and various between 20Mio/s and 100Mio/s even if the client is 2 or 3 links bonded.

I tried to connect a client to the SAN directly (without using a switch) and I still have the problems.

Thanks for guiding me

Hi,

For me, you can't aggregate bandwith with bonding.
You have to use MPIO.

Since there could be few things causing lowered performance except hardware failure so the best thing to do first is to download logs from DSS (in Web GUI -> Status -> Hardware -> Logs) and search tests.log file for:
1) hdparm - to check actual raid performance
2) ethtool - to check actual network cards speed
3) ifconfig - to see if there is any packet loss (errors in network transmission)
Please take a look at it cause it may be for example cards are running only 100MBps mode for some reason.

Also you did not tell us what is the build of DSSv6 that you are running now?

I did the test :
*-----------------------------------------------------------------------------*
hdparm -t /dev/sda
*-----------------------------------------------------------------------------*


/dev/sda:
Timing buffered disk reads: 774 MB in 3.00 seconds = 257.96 MB/sec

(Same results on sdb, sdbc, sdd)

*-----------------------------------------------------------------------------*
hdparm -t /dev/sde
*-----------------------------------------------------------------------------*


/dev/sde:
Timing buffered disk reads: 400 MB in 3.00 seconds = 133.25 MB/sec

This is the last SSD, it is a bit slow !

*-----------------------------------------------------------------------------*
hdparm -t /dev/sdf
*-----------------------------------------------------------------------------*


/dev/sdf:
Timing buffered disk reads: 274 MB in 3.02 seconds = 90.81 MB/sec

This is the 2x 2.5" RAID1 HDD used for DSS installation

*-----------------------------------------------------------------------------*
hdparm -t /dev/md0
*-----------------------------------------------------------------------------*


/dev/md0:
Timing buffered disk reads: 1884 MB in 3.00 seconds = 627.64 MB/sec

It's a bit slow for a Raid5 of 5 SSD, maybe the problem is caused by sde...

*-----------------------------------------------------------------------------*
ethtool bond0
*-----------------------------------------------------------------------------*

Settings for bond0:
No data available

It that normal?

All eth0,eth1,eth2 are set to 1000Mbit/s and no errors occured.


Version of DSS : 6.0up75.8401.5377 64bit

You need a small update for your NIC card, as you are having build up75, so please open a support ticket, and ask me to send you a Small Update files.
0852-DSS_V6_up75-Intel_1Gbps_NIC_driver_set
0853-DSS_V6_up75-Intel_10Gbps_NIC_driver_set

______________________
Alaa Souqi

I'm still using the Lite version since I don't know if DSS will works well with my configuration, so I can't open a support ticket.
What should I do ? Pay for support for obtaining a patch who will maybe not solves my problems?
Thanks however

search for individual cards with ethtool, not for bond, like ethtool eth0 etc.

*-----------------------------------------------------------------------------*
ethtool eth0
*-----------------------------------------------------------------------------*

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbag
Wake-on: g
Current message level: 0x00000001 (1)
Link detected: yes

*-----------------------------------------------------------------------------*
ethtool eth1
*-----------------------------------------------------------------------------*

Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

*-----------------------------------------------------------------------------*
ethtool eth2
*-----------------------------------------------------------------------------*

Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

*-----------------------------------------------------------------------------*
ifconfig -a
*-----------------------------------------------------------------------------*

bond0 Link encap:Ethernet HWaddr 02:4B:F6:97:7C:1F
inet addr:188.XXX.XXX.XXX Bcast:188.XXX.XXX.XXX Mask:255.255.255.240
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2275004 errors:0 dropped:0 overruns:0 frame:0
TX packets:4133574 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:157889545 (150.5 MiB) TX bytes:6054399402 (5.6 GiB)

eth0 Link encap:Ethernet HWaddr 02:4B:F6:97:7C:1F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:745161 errors:0 dropped:0 overruns:0 frame:0
TX packets:1324506 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:53462615 (50.9 MiB) TX bytes:1958278490 (1.8 GiB)

eth1 Link encap:Ethernet HWaddr 02:4B:F6:97:7C:1F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:145475 errors:0 dropped:0 overruns:0 frame:0
TX packets:1867 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9769126 (9.3 MiB) TX bytes:232141 (226.7 KiB)

eth2 Link encap:Ethernet HWaddr 02:4B:F6:97:7C:1F
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1384368 errors:0 dropped:0 overruns:0 frame:0
TX packets:2807201 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:94657804 (90.2 MiB) TX bytes:4095888771 (3.8 GiB)

ipddp0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
BROADCAST NOARP MULTICAST MTU:585 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2093649 errors:0 dropped:0 overruns:0 frame:0
TX packets:2093649 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:338331735 (322.6 MiB) TX bytes:338331735 (322.6 MiB)

So far looks fine, next steps to determine what is slowing everything down may be:
- disable bonding and use single Ethernet controller
- test the system without s/w raid, check if raid has completed initialization, which is a huge load for a cpu
- make sure option WB (WriteBack caching) is enabled when assigning volume to iSCSI target, which will slightly speed up data writing

After testing few possibilities you should be able to find the weakest link.