Resolving a split brain on replication

[enealDC]

Hi Guys. I have a problem with two nodes where I cannot reestablish replication. I've tried almost everything. I've removed the contents of units on both nodes and nothing works. Here is what is being logged on the primary:

block drbd0: sock_recvmsg returned -110
block drbd0: peer( Secondary -> Unknown ) conn( WFBitMapS -> BrokenPipe ) pdsk( UpToDate -> DUnknown )
block drbd0: short read expecting header on sock: r=-110
block drbd0: sock_sendmsg returned -32
block drbd0: short sent ReportBitMap size=4096 sent=2904
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: Connection closed
block drbd0: conn( BrokenPipe -> Unconnected )
block drbd0: receiver terminated
block drbd0: Restarting receiver thread
block drbd0: receiver (re)started
block drbd0: conn( Unconnected -> WFConnection )
block drbd0: Handshake successful: Agreed network protocol version 91
block drbd0: conn( WFConnection -> WFReportParams )
block drbd0: Starting asender thread (from drbd0_receiver [26312])
block drbd0: data-integrity-alg: <not-used>
block drbd0: drbd_sync_handshake:
block drbd0: self E32B7128246F202F:E5E9F5BDC1092F1F:0000000000000004 :0000000000000000 bits:326677276 flags:0
block drbd0: peer 4D573A2961561D98:3D7D2ECABFAA8F87:0000000000000004 :0000000000000000 bits:326677275 flags:0
block drbd0: uuid_compare()=-100 by rule 100
block drbd0: Split-Brain detected, dropping connection!
block drbd0: helper command: /sbin/drbdadm split-brain minor-0
block drbd0: meta connection shut down by peer.
block drbd0: conn( WFReportParams -> NetworkFailure )
block drbd0: asender terminated
block drbd0: Terminating asender thread
block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
block drbd0: conn( NetworkFailure -> Disconnecting )
block drbd0: error receiving ReportState, l: 4!
block drbd0: Connection closed
block drbd0: conn( Disconnecting -> StandAlone )
block drbd0: receiver terminated

This doesn't seem resolvable from any of the options present in the gui or the console. We have removed contents of units, restored factory configuration and setup.. The results are the same. Any thoughts?

[Gr-R]

make sure nodes can ping each other.

clear metadata on both nodes, and verify source/destination settings.

reestablish connections and then the add the task.

[enealDC]

I've already done this. The nodes can see each other plainly. I've practically redone the configuration from scratch and the result is always the same.
We have also replicating from the secondary node to the first node.

[Gr-R]

check and make sure NIC settings are same on both nodes... jumbo frames ?

also make sure volumes are same size.

have you done anything to tune drdb ? same settings on both nodes ?

[enealDC]

Keep in mind that as I've already said, I have completely restored factory configuration and setup. Removed contents of units, etc. These units are on the same vlan and when you configure replication, you can see the volume on the other end indicating good connectivity. In addition, the task gets created on the remote node..
As I said previously, it does not appear that this error or situation is resolvable via any of the interfaces I have at my disposal..

[Gr-R]

send me the logs from each node, via support, or in PM

[enealDC]

sent to support@open-e.com