Visit Open-E website
Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: 802.3ad bonding Intel X520-DA2 with direct attached doesn't work

  1. #1

    Default 802.3ad bonding Intel X520-DA2 with direct attached doesn't work

    Hello !

    Just setting up a new HA pair of DSS V7.
    We're using Intel X520-DA2 dual port cards with fiber optics transceiver
    (of course original Intel E10GSFPSR, the card will not work with other transceivers) for replication.
    The adapters are directly (switchless) connected with 7m fiber optics cable OM3, jumbo frames enabled
    with mtu 9000 and both adapters have connectivity, pings are ok.

    When we bond the two interfaces with 802.3ad, there is no more ip-connectivity,
    although the system tells us "bond0: link defintily up....".
    Ping from console fails with "destination host unreachable".

    We are aware that we will not gain replication performance by bonding,
    this should bond only due to HA reasons.

    Here comes the dmesg:
    Code:
    [   49.569239] bonding: Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
    [   49.569242] bonding: MII link monitoring set to 50 ms
    [   49.569698] bonding: bond0 is being deleted...
    [   49.798649] bonding: bond0 is being created...
    [   49.799036] bonding: bond0: setting mode to 802.3ad (4).
    [   49.799083] bonding: bond0: Setting MII monitoring interval to 50.
    [   50.295547] bonding: bond0: enslaving eth1 as a backup interface with a down link.
    [   50.383980] ixgbe 0000:02:00.0: eth1: detected SFP+: 5
    [   50.663571] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   50.764830] bonding: bond0: enslaving eth3 as a backup interface with a down link.
    [   50.766114] bonding: bond0: Unable to set primary slave; bond0 is in mode 4
    [   50.766121] bonding: bond0: link status definitely up for interface eth1.
    [   50.766124] bonding: bond0: link status definitely up for interface eth3.
    [   50.822454] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   50.843234] ixgbe 0000:02:00.1: eth3: detected SFP+: 6
    [   51.112841] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   51.121983] bonding: bond0: link status definitely up for interface eth3.
    [   53.846653] ixgbe 0000:02:00.0: eth1: NIC Link is Down
    [   53.869008] bonding: bond0: link status definitely down for interface eth1, disabling it
    [   54.056359] ixgbe 0000:02:00.1: eth3: NIC Link is Down
    [   54.067512] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   55.685567] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   58.392232] igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   61.986689] igb: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
    [   64.302288] igb: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   65.699760] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   65.721518] bonding: bond0: link status definitely up for interface eth1.
    [   66.298843] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   66.318821] bonding: bond0: link status definitely up for interface eth3.
    [   72.752510] ixgbe 0000:02:00.1: eth3: changing MTU from 1500 to 9000
    [   73.340577] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   73.343700] ixgbe 0000:02:00.0: eth1: changing MTU from 1500 to 9000
    [   73.399141] ixgbe 0000:02:00.1: eth3: detected SFP+: 6
    [   73.888685] bonding: bond0: link status definitely down for interface eth1, disabling it
    [   73.958458] ixgbe 0000:02:00.0: eth1: detected SFP+: 5
    [   74.181954] Loading iSCSI transport class v2.0-871.
    [   74.192906] iscsi: registered transport (tcp)
    [   74.666148] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   74.759576] bonding: bond0: link status definitely up for interface eth1.
    [   74.786001] bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
    [   75.106518] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   75.166044] bonding: bond0: link status definitely up for interface eth3.
    [   77.704366] EXT3-fs: barriers not enabled
    [   77.735366] kjournald starting.  Commit interval 5 seconds
    [   77.735583] EXT3-fs (dm-15): using internal journal
    [   77.735588] EXT3-fs (dm-15): mounted filesystem with ordered data mode
    What (and why) is that warning at 74.786001 ?

    Any suggestions ?

    Regards,
    Ralph

  2. #2
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
    LACP bond requires a switch that supports LACP bonding.

  3. #3

    Default

    Uhm, we are using 802.3ad bonds with direct attached Intel 1GBit cards and DSS v6
    without any problem for ages now. Never needed a switch...

    Are You sure ?

    Regards,
    Ralph

  4. #4
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    I could be wrong, but I think the 10GBe needs switch to do it.

  5. #5

    Default

    Taken from http://www.kernel.org/doc/Documentat...ng/bonding.txt


    5. Switch Configuration
    =======================
    For this section, "switch" refers to whatever system the bonded devices are directly connected to (i.e., where the other end of the cable plugs into). This may be an actual dedicated switch device, or it may be another regular system (e.g., another computer running Linux),

    The active-backup, balance-tlb and balance-alb modes do not require any specific configuration of the switch.

    The 802.3ad mode requires that the switch have the appropriate ports configured as an 802.3ad aggregation. The precise method used to configure this varies from switch to switch, but, for example, a Cisco 3550 series switch requires that the appropriate ports first be grouped together in a single etherchannel instance, then that etherchannel is set to mode "lacp" to enable 802.3ad (instead of standard EtherChannel).
    So "switch" could be whatever (my understanding...).

    Regards,
    Ralph

  6. #6
    Join Date
    May 2008
    Location
    Hamburg, Germany
    Posts
    108

    Default

    Hi Ralph,

    what's in the logs of the other server? Does it look similar?

    802.3ad between two Linux servers, directly connected, should work (as long as you have only a single bond per server - there's no way to set partner IDs with Linux ).

    There's a lot of ups & downs of the interfaces going on during boot - I take it that's caused by setting the interface parameters. This happens after bonding the interfaces, maybe that's confusing the bonding driver?

    I suggest that you start both servers from a Linux boot medium and try to set up the LACP link manually. That way you can confirm that it indeed works and can spot potential pitfalls.

    Regards,
    jmo

  7. #7

    Default

    Great stuff !

    Bonding works, after deleting bond on both systems, reboot both systems and then
    recreate the bond.
    Those up/downs were caused by booting both systems simultanously.
    Maybe that would prevent bond from coming up correctly...
    Just performing a reboot of ONE system now to see, wether it's a general reboot issue:

    And it is.

    Rebooting one system causes the bond to fail.

    Works again when we remove bonding and recreate the bond on the booted system via webgui.
    Also when we setup jumbo frames to 9000 on both systems via console while bond is running,
    we are losing connectivity.
    Only way to get it back ist remove bond and recreate via gui, this time on both systems.
    Unfortunately then our jumboframes are gone to 1500. So no way to use jumbos.
    Edit: And no proper reboot of the system also....

    Regards,
    Ralph
    Last edited by DIESOFTWARE-RS; 09-06-2012 at 07:34 PM.

  8. #8
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    set jumbo frames on each NIC, then create the bond, and set the frame size for the bond.

  9. #9

    Default

    Quote Originally Posted by Gr-R View Post
    set jumbo frames on each NIC, then create the bond, and set the frame size for the bond.
    Yep, we have done this before...
    (without success...)

    Regards
    Last edited by DIESOFTWARE-RS; 09-07-2012 at 09:54 AM.

  10. #10

    Default

    Should we open a ticket regarding this ?

    regards
    Ralph

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •