Bonding 802.3ad betting performance ?

Hi

I'm trying to get better performance for my iSCSI traffic.

I activate Jumbo Frame on all servers and MPIO in Vmware ESXi 4 with Round Robin Policy for Paths.

Now i'm focus on OpenE DSS.

I tried 802.3ad and i'm unable to get this working.

Switch is a Linksys SLM2008, my bonding goes up in 802.3ad but some server can't access to the OpenE. If i unplug a cable of the bond or restart the switch it's another server wich lost access to OpenE.

So is there someone here using 802.3ad with OpenE and getting really performance of 2Gbit link.

Thanks

I've seen some switches that claim to support 802.3, but do not
try going straight to the esx server from your dss to see if you get better performance.

you speed will not increase with bonding, it will give you more throughput.
something like having 2 water faucets turned on. the pressure (speed) is still 80Psi
but you will have twice (bonding) the water flow.

you need to push a lot of data to test

I'll try another switch then.

I don't want to connect directly ESX to OpenE because i have two OpenE server and i want to use Ip Failover and HA.

Here some bench with CrystalDiskMark 2.2 in a Windows 2008 Server VM on my ESXi (MPIO on 2 path RR Mode from Vmware) and Balance-ALB for OpenE. Jumbo Frame are up everywhere (Switch, OpenE, Vmware)

Without synchronous Replication Tasks (Link between OpenE are only 1Gbit):

Sequential Read : 101.344 MB/s
Sequential Write : 97.572 MB/s
Random Read 512KB : 95.121 MB/s
Random Write 512KB : 86.132 MB/s
Random Read 4KB : 14.091 MB/s
Random Write 4KB : 8.158 MB/s

Test Size : 100 MB

With synchronous Replication Tasks (Link between OpenE are only 1Gbit):

Sequential Read : 102.383 MB/s
Sequential Write : 80.400 MB/s
Random Read 512KB : 97.688 MB/s
Random Write 512KB : 67.150 MB/s
Random Read 4KB : 13.703 MB/s
Random Write 4KB : 3.384 MB/s

Test Size : 100 MB


I tried other bonding on OpenE without success :
- Balance-RR give me only 50MB/s in Read Performance for better Write...
- 802.3ad is not working

For me a good setup will be something like :
- ESXi using MPIO and 2 NIC connected to an ISCSI Switch
- OpenE connected to the ISCSI switch with 802.3ad mode 2 NIC
- Replication between OpenE on a dedicated LAN with 802.3ad mode and 2 NIC (may be 2 direct cable in 802.3ad?)

Is this possible ?

802.3ad bonding has been discussed several times on this forum. As nsc said, you will not get any performance boost for a single host connection via 802.3ad, since the switch will just route all the traffic from one IP on the left side, to the other IP on the right side through the same connections. In case the link where the actual traffic goes through fails, it will re-route to the remaining link(s).

If you need higher single-connections speeds, you will have to go with 10GbE. Otherwise you will have to have multiple servers (likely with odd and even IPs) connected to the switch and your Open-E DSS to gain a overall throughput boost. One connection cannot exceed 1Gbis/s with 802.3ad bonding.

Cheers,
budy

Hi Buddy,

Have you any info about the cost a switch 10GB and 3 Network Cards 10GB ?

Both compatible with OpenE of course ;-)

Thanks

hi,
we have succesfully setup our systems with a 10GB CX4 Network
dual port Intel 10GB CX4 PCI-E V2 ~ 600€
6 Port HP 10GB CX4 switch ~ 4000 €

CX4 is actually the cheapest way to implement 10GB.

greetings
roger

Thanks a lot for this info :)

Quote:

---

Originally Posted by nsc

I'll try another switch then.

I don't want to connect directly ESX to OpenE because i have two OpenE server and i want to use Ip Failover and HA.

Here some bench with CrystalDiskMark 2.2 in a Windows 2008 Server VM on my ESXi (MPIO on 2 path RR Mode from Vmware) and Balance-ALB for OpenE. Jumbo Frame are up everywhere (Switch, OpenE, Vmware)

Without synchronous Replication Tasks (Link between OpenE are only 1Gbit):

Sequential Read : 101.344 MB/s
Sequential Write : 97.572 MB/s
Random Read 512KB : 95.121 MB/s
Random Write 512KB : 86.132 MB/s
Random Read 4KB : 14.091 MB/s
Random Write 4KB : 8.158 MB/s

Test Size : 100 MB

With synchronous Replication Tasks (Link between OpenE are only 1Gbit):

Sequential Read : 102.383 MB/s
Sequential Write : 80.400 MB/s
Random Read 512KB : 97.688 MB/s
Random Write 512KB : 67.150 MB/s
Random Read 4KB : 13.703 MB/s
Random Write 4KB : 3.384 MB/s

Test Size : 100 MB


I tried other bonding on OpenE without success :
- Balance-RR give me only 50MB/s in Read Performance for better Write...
- 802.3ad is not working

For me a good setup will be something like :
- ESXi using MPIO and 2 NIC connected to an ISCSI Switch
- OpenE connected to the ISCSI switch with 802.3ad mode 2 NIC
- Replication between OpenE on a dedicated LAN with 802.3ad mode and 2 NIC (may be 2 direct cable in 802.3ad?)

Is this possible ?

---

100MB/s thats low for rr-mpio, im getting around 170-180MB/s read and about 100MB/s write (replication over 1gbit), my setup: xen, 2 gbit switches and 2 dss boxes

what is your bond config in OpenE ?

I tried Balance RR, Balance-ALB & 802.3ad.

In Balance RR i get better write but lower Read...

May be Xen do better...

We have had real good luck with Supermicro Dual Port 10g low profile cards. About $450.00 for the model AOC-STG-I2. Based on Intel 10g chip and fully supported by Open-E. Haven't found a less expensive supported dual port 10g card.

But CX4 is limited to 15m right ? Looks like it will be too short for me (my two san are not on the same floor).

On 10GBase-T it's 100m with copper but i can't find a switch 10GBase-T (8 ports?) and NIC 10GBase-T compatible with OpenE.

you can buy extender for cx4 from intel
with intel connect cable you can go up to 100m

Quote:

---

Originally Posted by nsc

what is your bond config in OpenE ?

I tried Balance RR, Balance-ALB & 802.3ad.

In Balance RR i get better write but lower Read...

May be Xen do better...

---

in my case replication goes trough 2x gbit balance rr bonding

bonding shouldnt impact read performance at all, since only writes are being replicated to the slave, read operations are being serverd direct from your master box, so if you are not getting any better then 1ge read performance then there is something wrong with your esx - dss setup.

and btw bonding will _not_ increase your replication/write throughput - you will need to change to infiniband or 10ge

I aggree with the fact that replication must not impact read performance.

My ESXi 4 is in MPIO support RR Vmware : i see activity on my two ETH for iISCSI traffic (but may it's not working...)

So what is your teaming mode in DSS to get 170Mb / s in read ?

I tried Balance-RR (lower read performance) and Balance ALB.

What is yours ?

Quote:

---

Originally Posted by nsc

I aggree with the fact that replication must not impact read performance.

My ESXi 4 is in MPIO support RR Vmware : i see activity on my two ETH for iISCSI traffic (but may it's not working...)

So what is your teaming mode in DSS to get 170Mb / s in read ?

I tried Balance-RR (lower read performance) and Balance ALB.

What is yours ?

---

bonding between client and dss? what for? mpio does it for me.
for an example: dss with 2x ge ports,
port A with 10.0.0.1/24 connected to switch A
port B with 10.0.1.1/24 connected to switch B
now client (xen in this case) also have 2ge ports, which are connected to switch A and B, this way each of the port is seperate from the other one and in its own network, for the 'balancing' i setup a iscsi connection to each of the network (10.0.0.1 and 10.0.1.1) and put mpio over it to balance the io requests (change path every 4io requests) - this way im getting about 170MB/s read

ok red now i have understand :)

I want to have HA Open-e so i have only one Virtual IP for link between client & OpenE... I can't use Mpio on IP for the link... (Or may be DSS V6 can give me more than one Virtual IP?

I'll try your setup anyway.

But with your setup you dont have automatic failover for your san right ?

Quote:

---

Originally Posted by nsc

ok red now i have understand :)

I want to have HA Open-e so i have only one Virtual IP for link between client & OpenE... I can't use Mpio on IP for the link... (Or may be DSS V6 can give me more than one Virtual IP?

---

Yes, you can have more than one vIP - I use MPIO with 2 vIPs.

i removed my bond for iscsi (eth3 / eth5)

on eth3 : 10.0.0.1 Jumbo Frame 9000
on eth5 : 10.0.0.3 Jumbo Frame 9000

on esxi host i add 10.0.0.3 for iscsi

now i have my 4 paths to access my iscsi volume

Round Robin Vmware is active

No perfomance improvement with CrystalDiskMark :(

I'm using OpenE V5 (not V6) may it's a limitation with IET / SCST option for ISCSI ?

I'm planning to move to V6...

i had to tweak mpio settings to get any increase in performance at all - 4 IOs per path instead of default 1000

i dont find this kind of option in Vmware.

Following a blog message, i just have to activate Round Robin option.

I'll wait for V6 then...

Quote:

---

Originally Posted by rogerk

you can buy extender for cx4 from intel
with intel connect cable you can go up to 100m

---

To be clear it is not an "extender" but cables which use ultra high quality cables combined with an amplifier built-in to the CX4 connector which allow for the signal distance to be extended to 100m.

I have a similar issue.
I have a Supermicro server with a 3ware Inc 9650SE SATA-II RAID PCIe with 8x SATA 1TB Drives setup in RAID6 with 2 x onboard 1Gb interfaces and 2 x dual 1Gb interfaces.
eth0: MGMT
eth1: Internet facing interface

Bond0: 172.16.31.200 (eth2-5 bonded in 802.3ad)
eth2: 192.168.1.1
eth3: 192.168.2.1
eth4: 192.168.3.1
eth5: 192.168.4.1

I then have 2 Supermicro 1U servers with 4 interfaces, they are as follows:
eth0: MGMT
eth1: Internet and Dot1q trunking
eth2 & eth3 are setup in a bond and connectet back to iSCSI on IP address 172.16.31.200

I have a linux and windows VM on each XenServer, the windows system running CrystalDiskMark 3.0 gets the following results:
Read:
Seq: 94.95
512K: 89.05
4K: 16.53
4K QD32: 67.09

Write:
Seq: 75.67
512K: 43.64
4K: 3.130
4K QD32: 3.461

Everything runs fine, though these slowish speeds are a bit of a concern. My issue is every now and then I have a problem where sites on the Linux VM's fail to load for 10-15 seconds then they all load fine. During this time if I have a file open via CLI using vi or nano and I try save it, it also waits for 10-15 seconds and then the file saves and I see my bash shell, at the same time all sites start loading.

I know this is not an internet problem as the hosts are all pingable at the time and I can still press enter and see my bash shell new line, it's just read/write seems to hang for 10-20 seconds from time to time.

I read in this thread that some people use something called mpio (never seen or heard of it) and connect to the multiple interfaces IP's rather than a single Bond0 IP address.

As this is a production system in a datacenter I'm reluctant to change too much incase I break it. Can someone with knowledge about iSCSI with opene & xenserver please make suggestions.

Thanks in advance
Barry Murphy

I have seen where the iSCSI initiators will drop the connections due to the fact of a slow response time from the Target or the speed, and if your doing some replications what is the speed of the volume replication link (please dont share this link w/ other services).

Or RAID could be doing something on the backend like RAID health check or disk or BBU check, slows the performance and if every one is hammering it could cause a delay but we would have to see the logs or check the 3Ware logs.

hi,

i found my old post, finally i found the trick in vmware thant i neved posted back here...

In fact, we just have to change the round robin policy (iops options). I assume that you have MPIO working (multiple connections from 1 host to 1 lun)
First, activate round robin in vmware (Manage Path on Datastore) then connect to ESX Host with SSH

list vmfs disk on esx host :

Code:

---

ls /vmfs/devices/disks/eui*
/vmfs/devices/disks/eui.65304753474c3552
/vmfs/devices/disks/eui.65304753474c3552:1

---

get current config:

Code:

---

/usr/sbin/esxcli nmp roundrobin getconfig --device eui.65304753474c3552
Byte Limit: 10485760
Device: eui.65304753474c3552
I/O Operation Limit: 1000
Limit Type: Default
Use Active Unoptimized Paths: false

---

modify iops option

Code:

---

/usr/sbin/esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device eui.65304753474c3552

---

verify option

Code:

---

/usr/sbin/esxcli nmp roundrobin getconfig --device eui.65304753474c3552
Byte Limit: 10485760
Device: eui.65304753474c3552
I/O Operation Limit: 1
Limit Type: Default
Use Active Unoptimized Paths: false

---

with this config done on all my Host for all my datastore i get 170 - 180Mps in read performance with 2X1Gbit.

Now my last question, can i have the same config with iscsi failover, i just need 2 virtual ip :)

Thanks

nsc

This is correct that you will need 2 VIP's to work with the MPIO.

I also find some information in here, thank all

Hello have you any info about the cost a switch 10GB and 3 Network Cards 10GB ?

Both compatible with OpenE of course ;-)

i also agree this thanks ....

____________________________________

Plc Training In Chennai

You can find many brands that we support for NIC cards and switchs such as:
Chelsio Communications, Intel, Myricom, NetEffect, Neterion, NetXen, SMC Networks, ... and many more.

Please visit our website at:
http://www.open-e.com/service-and-su...tibility-list/

to find compatibility list ( if you have a brand/model that is not listed, please ask our support team to help you more about that ).

For the prices, you can check many websites that sell switch/NIC online or contact a reseller near you, or if some or our forum reader help with that :)