Resolving a split brain on replication

Printable View

03-24-2011, 01:46 AM
enealDC

Resolving a split brain on replication

Hi Guys. I have a problem with two nodes where I cannot reestablish replication. I've tried almost everything. I've removed the contents of units on both nodes and nothing works. Here is what is being logged on the primary:

block drbd0: sock_recvmsg returned -110
block drbd0: peer( Secondary -> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
block drbd0: short read expecting header on sock: r=-110
block drbd0: sock_sendmsg returned -32
block drbd0: short sent ReportBitMap size=4096 sent=2904
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: Connection closed
block drbd0: conn( BrokenPipe -> Unconnected )
block drbd0: receiver terminated
block drbd0: Restarting receiver thread
block drbd0: receiver (re)started
block drbd0: conn( Unconnected -> WFConnection )
block drbd0: Handshake successful: Agreed network protocol version 91
block drbd0: conn( WFConnection -> WFReportParams )
block drbd0: Starting asender thread (from drbd0_receiver [26312])
block drbd0: data-integrity-alg: <not-used>
block drbd0: drbd_sync_handshake:
block drbd0: self E32B7128246F202F:E5E9F5BDC1092F1F:0000000000000004 :0000000000000000 bits:326677276 flags:0
block drbd0: peer 4D573A2961561D98:3D7D2ECABFAA8F87:0000000000000004 :0000000000000000 bits:326677275 flags:0
block drbd0: uuid_compare()=-100 by rule 100
block drbd0: Split-Brain detected, dropping connection!
block drbd0: helper command: /sbin/drbdadm split-brain minor-0
block drbd0: meta connection shut down by peer.
block drbd0: conn( WFReportParams -> NetworkFailure )
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
block drbd0: conn( NetworkFailure -> Disconnecting )
block drbd0: error receiving ReportState, l: 4!
block drbd0: Connection closed
block drbd0: conn( Disconnecting -> StandAlone )
block drbd0: receiver terminated

This doesn't seem resolvable from any of the options present in the gui or the console. We have removed contents of units, restored factory configuration and setup.. The results are the same. Any thoughts?
03-24-2011, 02:51 AM
Gr-R

make sure nodes can ping each other.

clear metadata on both nodes, and verify source/destination settings.

reestablish connections and then the add the task.
03-24-2011, 11:52 AM
enealDC

I've already done this. The nodes can see each other plainly. I've practically redone the configuration from scratch and the result is always the same.
We have also replicating from the secondary node to the first node.
03-24-2011, 11:58 AM
Gr-R

check and make sure NIC settings are same on both nodes... jumbo frames ?

also make sure volumes are same size.

have you done anything to tune drdb ? same settings on both nodes ?
03-24-2011, 11:58 AM
enealDC

Keep in mind that as I've already said, I have completely restored factory configuration and setup. Removed contents of units, etc. These units are on the same vlan and when you configure replication, you can see the volume on the other end indicating good connectivity. In addition, the task gets created on the remote node..
As I said previously, it does not appear that this error or situation is resolvable via any of the interfaces I have at my disposal..
03-24-2011, 12:14 PM
Gr-R

send me the logs from each node, via support, or in PM
03-24-2011, 02:03 PM
enealDC

sent to support@open-e.com
03-24-2011, 04:29 PM
Gr-R

there are sock errors on the network connection between the nodes.

===
block drbd0: [drbd0_worker/21962] sock_sendmsg time expired, ko = 4294967142
block drbd0: sock_recvmsg returned -110
block drbd0: sock_sendmsg returned -32
block drbd0: peer( Secondary -> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
block drbd0: short read expecting header on sock: r=-110
block drbd0: short sent ReportBitMap size=4096 sent=1504
===

reset the DRDB configuration to default and see if it works

default values:
max-buffers=2048
max-epoch-size=2048
unplug-watermark=128
sndbuf-size=0
al-extents=127
no-disk-barrier=off
no-disk-flushes=off
03-24-2011, 07:32 PM
enealDC

Thanks for looking it over.
We started with the defaults and modified it as a last resort to see if it would fix it.
In short, we've already tried reestablishing it using the default settings..
03-24-2011, 07:34 PM
Gr-R

Quote:

Originally Posted by enealDC

Thanks for looking it over.
We started with the defaults and modified it as a last resort to see if it would fix it.
In short, we've already tried reestablishing it using the default settings..

OK, I will dig deeper
03-24-2011, 08:11 PM
Gr-R

Check the MTU on the NICs on both nodes:
eth0 at 1500 is OK, but the others need to be consistant. [9000]

san01
eth0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
eth1 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth2 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth3 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth4 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth5 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9000 Metric:1

san02
eth0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
eth1 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth2 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth3 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth4 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
eth5 UP BROADCAST RUNNING SLAVE MULTICAST MTU:9216 Metric:1
03-25-2011, 12:43 AM
enealDC

great find and good attention to detail! it even makes more sense because both bond0 and bond1 are in the same subnet so a [jumbo] frame could be sent from bond1 on san01 and be received by bond0 on san02. And because the default frame size for ping is 1500 mtu, that makes it seem like connectivity is good!
i'm going to make those changes and see if they fix it. i'm pretty confident at this point!
03-25-2011, 01:39 AM
Gr-R

Quote:

Originally Posted by enealDC

great find and good attention to detail! it even makes more sense because both bond0 and bond1 are in the same subnet so a [jumbo] frame could be sent from bond1 on san01 and be received by bond0 on san02. And because the default frame size for ping is 1500 mtu, that makes it seem like connectivity is good!
i'm going to make those changes and see if they fix it. i'm pretty confident at this point!

Let us know.
Hope it works, I confident it will ;)
03-25-2011, 09:09 PM
enealDC

Thanks! That was it.