802.3ad bonding Intel X520-DA2 with direct attached doesn't work

Printable View

09-06-2012, 12:20 PM
DIESOFTWARE-RS

802.3ad bonding Intel X520-DA2 with direct attached doesn't work

Hello !

Just setting up a new HA pair of DSS V7.
We're using Intel X520-DA2 dual port cards with fiber optics transceiver
(of course original Intel E10GSFPSR, the card will not work with other transceivers) for replication.
The adapters are directly (switchless) connected with 7m fiber optics cable OM3, jumbo frames enabled
with mtu 9000 and both adapters have connectivity, pings are ok.

When we bond the two interfaces with 802.3ad, there is no more ip-connectivity,
although the system tells us "bond0: link defintily up....".
Ping from console fails with "destination host unreachable".

We are aware that we will not gain replication performance by bonding,
this should bond only due to HA reasons.

Here comes the dmesg:

Code:

[ 49.569239] bonding: Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) [ 49.569242] bonding: MII link monitoring set to 50 ms [ 49.569698] bonding: bond0 is being deleted... [ 49.798649] bonding: bond0 is being created... [ 49.799036] bonding: bond0: setting mode to 802.3ad (4). [ 49.799083] bonding: bond0: Setting MII monitoring interval to 50. [ 50.295547] bonding: bond0: enslaving eth1 as a backup interface with a down link. [ 50.383980] ixgbe 0000:02:00.0: eth1: detected SFP+: 5 [ 50.663571] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 50.764830] bonding: bond0: enslaving eth3 as a backup interface with a down link. [ 50.766114] bonding: bond0: Unable to set primary slave; bond0 is in mode 4 [ 50.766121] bonding: bond0: link status definitely up for interface eth1. [ 50.766124] bonding: bond0: link status definitely up for interface eth3. [ 50.822454] bonding: bond0: link status definitely down for interface eth3, disabling it [ 50.843234] ixgbe 0000:02:00.1: eth3: detected SFP+: 6 [ 51.112841] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 51.121983] bonding: bond0: link status definitely up for interface eth3. [ 53.846653] ixgbe 0000:02:00.0: eth1: NIC Link is Down [ 53.869008] bonding: bond0: link status definitely down for interface eth1, disabling it [ 54.056359] ixgbe 0000:02:00.1: eth3: NIC Link is Down [ 54.067512] bonding: bond0: link status definitely down for interface eth3, disabling it [ 55.685567] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 58.392232] igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 61.986689] igb: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 64.302288] igb: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 65.699760] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 65.721518] bonding: bond0: link status definitely up for interface eth1. [ 66.298843] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 66.318821] bonding: bond0: link status definitely up for interface eth3. [ 72.752510] ixgbe 0000:02:00.1: eth3: changing MTU from 1500 to 9000 [ 73.340577] bonding: bond0: link status definitely down for interface eth3, disabling it [ 73.343700] ixgbe 0000:02:00.0: eth1: changing MTU from 1500 to 9000 [ 73.399141] ixgbe 0000:02:00.1: eth3: detected SFP+: 6 [ 73.888685] bonding: bond0: link status definitely down for interface eth1, disabling it [ 73.958458] ixgbe 0000:02:00.0: eth1: detected SFP+: 5 [ 74.181954] Loading iSCSI transport class v2.0-871. [ 74.192906] iscsi: registered transport (tcp) [ 74.666148] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 74.759576] bonding: bond0: link status definitely up for interface eth1. [ 74.786001] bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond [ 75.106518] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 75.166044] bonding: bond0: link status definitely up for interface eth3. [ 77.704366] EXT3-fs: barriers not enabled [ 77.735366] kjournald starting. Commit interval 5 seconds [ 77.735583] EXT3-fs (dm-15): using internal journal [ 77.735588] EXT3-fs (dm-15): mounted filesystem with ordered data mode

What (and why) is that warning at 74.786001 ?

Any suggestions ?

Regards,
Ralph
09-06-2012, 01:53 PM
Gr-R

Quote:

bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond

LACP bond requires a switch that supports LACP bonding.
09-06-2012, 03:08 PM
DIESOFTWARE-RS

Uhm, we are using 802.3ad bonds with direct attached Intel 1GBit cards and DSS v6
without any problem for ages now. Never needed a switch...

Are You sure ?

Regards,
Ralph
09-06-2012, 03:23 PM
Gr-R

I could be wrong, but I think the 10GBe needs switch to do it.
09-06-2012, 04:08 PM
DIESOFTWARE-RS

Taken from http://www.kernel.org/doc/Documentat...ng/bonding.txt

Quote:

5. Switch Configuration
=======================
For this section, "switch" refers to whatever system the bonded devices are directly connected to (i.e., where the other end of the cable plugs into). This may be an actual dedicated switch device, or it may be another regular system (e.g., another computer running Linux),

The active-backup, balance-tlb and balance-alb modes do not require any specific configuration of the switch.

The 802.3ad mode requires that the switch have the appropriate ports configured as an 802.3ad aggregation. The precise method used to configure this varies from switch to switch, but, for example, a Cisco 3550 series switch requires that the appropriate ports first be grouped together in a single etherchannel instance, then that etherchannel is set to mode "lacp" to enable 802.3ad (instead of standard EtherChannel).

So "switch" could be whatever (my understanding...).

Regards,
Ralph
09-06-2012, 05:25 PM
jmo

Hi Ralph,

what's in the logs of the other server? Does it look similar?

802.3ad between two Linux servers, directly connected, should work (as long as you have only a single bond per server - there's no way to set partner IDs with Linux :( ).

There's a lot of ups & downs of the interfaces going on during boot - I take it that's caused by setting the interface parameters. This happens after bonding the interfaces, maybe that's confusing the bonding driver?

I suggest that you start both servers from a Linux boot medium and try to set up the LACP link manually. That way you can confirm that it indeed works and can spot potential pitfalls.

Regards,
jmo
09-06-2012, 06:26 PM
DIESOFTWARE-RS

Great stuff !

Bonding works, after deleting bond on both systems, reboot both systems and then
recreate the bond.
Those up/downs were caused by booting both systems simultanously.
Maybe that would prevent bond from coming up correctly...
Just performing a reboot of ONE system now to see, wether it's a general reboot issue:

And it is.

Rebooting one system causes the bond to fail.

Works again when we remove bonding and recreate the bond on the booted system via webgui.
Also when we setup jumbo frames to 9000 on both systems via console while bond is running,
we are losing connectivity.
Only way to get it back ist remove bond and recreate via gui, this time on both systems.
Unfortunately then our jumboframes are gone to 1500. So no way to use jumbos.
Edit: And no proper reboot of the system also....

Regards,
Ralph
09-06-2012, 06:30 PM
Gr-R

set jumbo frames on each NIC, then create the bond, and set the frame size for the bond.
09-06-2012, 06:35 PM
DIESOFTWARE-RS

Quote:

Originally Posted by Gr-R

set jumbo frames on each NIC, then create the bond, and set the frame size for the bond.

Yep, we have done this before...
(without success...)

Regards
09-07-2012, 04:55 PM
DIESOFTWARE-RS

Should we open a ticket regarding this ?

regards
Ralph
09-07-2012, 04:57 PM
Gr-R

You can, and I will get the research team to investigate. Reference this thread in your case.
09-07-2012, 05:01 PM
DIESOFTWARE-RS

We'll do on monday. It's about weekend now...

Have a nice one !

regards,
Ralph
09-12-2012, 01:29 PM
DIESOFTWARE-RS

Been quite busy, created ticket today, ticket number #1031370

regards,
Ralph
09-12-2012, 03:39 PM
DIESOFTWARE-RS

Support provided a small update to fix the issue, problem solved.

That was QUICK ! :-)

Thanks to Alaa from support !

regards,
Ralph
01-22-2013, 09:15 AM
bodo

Quote:

Originally Posted by DIESOFTWARE-RS

Support provided a small update to fix the issue, problem solved.

I'm curious about the solution: we also have a similar problem with two bonded intel igb-NICs (copper I350) that are directly connected (i.e. without a switch in between). We turned off autonegotation but that didn't help for a long time.

Can someone let us know how this problem was fixed, please?

thanks in advance
Bodo