Visit Open-E website
Results 1 to 10 of 15

Thread: 802.3ad bonding Intel X520-DA2 with direct attached doesn't work

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1

    Default 802.3ad bonding Intel X520-DA2 with direct attached doesn't work

    Hello !

    Just setting up a new HA pair of DSS V7.
    We're using Intel X520-DA2 dual port cards with fiber optics transceiver
    (of course original Intel E10GSFPSR, the card will not work with other transceivers) for replication.
    The adapters are directly (switchless) connected with 7m fiber optics cable OM3, jumbo frames enabled
    with mtu 9000 and both adapters have connectivity, pings are ok.

    When we bond the two interfaces with 802.3ad, there is no more ip-connectivity,
    although the system tells us "bond0: link defintily up....".
    Ping from console fails with "destination host unreachable".

    We are aware that we will not gain replication performance by bonding,
    this should bond only due to HA reasons.

    Here comes the dmesg:
    Code:
    [   49.569239] bonding: Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
    [   49.569242] bonding: MII link monitoring set to 50 ms
    [   49.569698] bonding: bond0 is being deleted...
    [   49.798649] bonding: bond0 is being created...
    [   49.799036] bonding: bond0: setting mode to 802.3ad (4).
    [   49.799083] bonding: bond0: Setting MII monitoring interval to 50.
    [   50.295547] bonding: bond0: enslaving eth1 as a backup interface with a down link.
    [   50.383980] ixgbe 0000:02:00.0: eth1: detected SFP+: 5
    [   50.663571] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   50.764830] bonding: bond0: enslaving eth3 as a backup interface with a down link.
    [   50.766114] bonding: bond0: Unable to set primary slave; bond0 is in mode 4
    [   50.766121] bonding: bond0: link status definitely up for interface eth1.
    [   50.766124] bonding: bond0: link status definitely up for interface eth3.
    [   50.822454] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   50.843234] ixgbe 0000:02:00.1: eth3: detected SFP+: 6
    [   51.112841] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   51.121983] bonding: bond0: link status definitely up for interface eth3.
    [   53.846653] ixgbe 0000:02:00.0: eth1: NIC Link is Down
    [   53.869008] bonding: bond0: link status definitely down for interface eth1, disabling it
    [   54.056359] ixgbe 0000:02:00.1: eth3: NIC Link is Down
    [   54.067512] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   55.685567] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   58.392232] igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   61.986689] igb: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
    [   64.302288] igb: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
    [   65.699760] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   65.721518] bonding: bond0: link status definitely up for interface eth1.
    [   66.298843] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   66.318821] bonding: bond0: link status definitely up for interface eth3.
    [   72.752510] ixgbe 0000:02:00.1: eth3: changing MTU from 1500 to 9000
    [   73.340577] bonding: bond0: link status definitely down for interface eth3, disabling it
    [   73.343700] ixgbe 0000:02:00.0: eth1: changing MTU from 1500 to 9000
    [   73.399141] ixgbe 0000:02:00.1: eth3: detected SFP+: 6
    [   73.888685] bonding: bond0: link status definitely down for interface eth1, disabling it
    [   73.958458] ixgbe 0000:02:00.0: eth1: detected SFP+: 5
    [   74.181954] Loading iSCSI transport class v2.0-871.
    [   74.192906] iscsi: registered transport (tcp)
    [   74.666148] ixgbe 0000:02:00.0: eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   74.759576] bonding: bond0: link status definitely up for interface eth1.
    [   74.786001] bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
    [   75.106518] ixgbe 0000:02:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
    [   75.166044] bonding: bond0: link status definitely up for interface eth3.
    [   77.704366] EXT3-fs: barriers not enabled
    [   77.735366] kjournald starting.  Commit interval 5 seconds
    [   77.735583] EXT3-fs (dm-15): using internal journal
    [   77.735588] EXT3-fs (dm-15): mounted filesystem with ordered data mode
    What (and why) is that warning at 74.786001 ?

    Any suggestions ?

    Regards,
    Ralph

  2. #2
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
    LACP bond requires a switch that supports LACP bonding.

  3. #3

    Default

    Uhm, we are using 802.3ad bonds with direct attached Intel 1GBit cards and DSS v6
    without any problem for ages now. Never needed a switch...

    Are You sure ?

    Regards,
    Ralph

  4. #4
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    I could be wrong, but I think the 10GBe needs switch to do it.

  5. #5

    Default

    Taken from http://www.kernel.org/doc/Documentat...ng/bonding.txt


    5. Switch Configuration
    =======================
    For this section, "switch" refers to whatever system the bonded devices are directly connected to (i.e., where the other end of the cable plugs into). This may be an actual dedicated switch device, or it may be another regular system (e.g., another computer running Linux),

    The active-backup, balance-tlb and balance-alb modes do not require any specific configuration of the switch.

    The 802.3ad mode requires that the switch have the appropriate ports configured as an 802.3ad aggregation. The precise method used to configure this varies from switch to switch, but, for example, a Cisco 3550 series switch requires that the appropriate ports first be grouped together in a single etherchannel instance, then that etherchannel is set to mode "lacp" to enable 802.3ad (instead of standard EtherChannel).
    So "switch" could be whatever (my understanding...).

    Regards,
    Ralph

  6. #6
    Join Date
    May 2008
    Location
    Hamburg, Germany
    Posts
    108

    Default

    Hi Ralph,

    what's in the logs of the other server? Does it look similar?

    802.3ad between two Linux servers, directly connected, should work (as long as you have only a single bond per server - there's no way to set partner IDs with Linux ).

    There's a lot of ups & downs of the interfaces going on during boot - I take it that's caused by setting the interface parameters. This happens after bonding the interfaces, maybe that's confusing the bonding driver?

    I suggest that you start both servers from a Linux boot medium and try to set up the LACP link manually. That way you can confirm that it indeed works and can spot potential pitfalls.

    Regards,
    jmo

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •