Errors From Replication

Printable View

07-22-2009, 02:58 PM
Diakiao

Errors From Replication

I keep getting replication errors mailed to me from my replication target SAN. It is located at a different branch via a 100MB link. The sync is asyncronous, and runs from 7PM-7AM the next morning. Here are the errors the SAN is e-mailing me in chronological order:

First e-mail:

2009/07/21 19:00:37 Replication:Volume Replication: Device lv0000: Connection (from WFConnection to WFBitMapT). Mode local/remote (from Secondary/Unknown to Secondary/Primary). DataStatus local/remote (from Consistent/DUnknown to Consistent/Consistent).

Second e-mail:

2009/07/21 19:00:37 Replication:Volume Replication: Device lv0000: Connection (from WFConnection to WFBitMapT). Mode local/remote (from Secondary/Unknown to Secondary/Primary). DataStatus local/remote (from Consistent/DUnknown to Consistent/Consistent).

2009/07/21 19:01:00 Replication:Volume Replication: Device lv0000: Connection (from WFBitMapT to SyncTarget). DataStatus local/remote (from Consistent/Consistent to Inconsistent/Consistent).

Third e-mail:

2009/07/22 07:00:36 Replication:Volume Replication: Device lv0000: Connection (from SyncTarget to WFConnection). Mode local/remote (from Secondary/Primary to Secondary/Unknown). DataStatus local/remote (from Inconsistent/Consistent to Consistent/DUnknown).

Fourth e-mail:

2009/07/22 07:00:36 Replication:Volume Replication: Device lv0000: Connection (from SyncTarget to WFConnection). Mode local/remote (from Secondary/Primary to Secondary/Unknown). DataStatus local/remote (from Inconsistent/Consistent to Consistent/DUnknown).

Now according to the task log on the source SAN the volumes were Consistent/Consistent at the end of the replication. But that last e-mail said unknown. Any ideas?
07-22-2009, 05:27 PM
To-M

I would check the test.log in the NIC section of ifconfig -a to see if there are any packet errors that might be generating those messages from consistent, inconsistent to unknown (Destination) to consistent.
07-22-2009, 07:05 PM
Diakiao

Well I just checked that. Here is a copy of the log from the source servers interface that connects to the second SAN. All the identifying info has been removed of course.

eth6 Link encap:Ethernet HWaddr
inet addr: Bcast: Mask:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1914064650 errors:0 dropped:0 overruns:0 frame:0
TX packets:3948037088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:209010884275 (194.6 GiB) TX bytes:5901393956807 (5.3 TiB)

Here is a copy of the log for the interface on the second SAN:

ncap:Ethernet HWaddr
inet addr: Bcast: Mask:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:372937392 errors:0 dropped:0 overruns:0 frame:0
TX packets:205165134 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:541926660966 (504.7 GiB) TX bytes:21169516242 (19.7 GiB)

So near as I can tell there are no collisions, errors, drops or anything.
07-22-2009, 07:30 PM
To-M

The only thing I can think of other then the NIC's is the RAID health that can cause this or there is more data that needs additional time other then the time frame that you have but in your case this does not apply as it looks like it was consistent.

Send in logs to support@open-e.com so the engineers can look into it.
07-22-2009, 09:40 PM
Diakiao

Done, I also sent a copy to your e-mail. If a support ticket needs to be opened or anything like that please let me know and I'll do it.

Thanks!
07-22-2009, 10:01 PM
To-M

I will see the ticket if it is sent to support@open-e.com
07-23-2009, 01:54 AM
To-M

What was the ticket # that you got from support?
07-23-2009, 01:12 PM
mhubert

Hi!

I get the same messages every day.
I have 4 async replication tasks and get 4 emails at start time and 4 at end time.
The task in the GUI shows consistent for every volume.

I thought, the messages are produced because the volume gets inconsistent/unknown as soon as the task stops?!

regards
Matthias
07-23-2009, 02:01 PM
budy

I am getting 8 of these each day. That's why I posted a feature request a while ago, since planned interruptions of a volume replication isn't really an error, but necessary:

http://forum.open-e.com/showthread.php?t=1394

Cheers,
budy
07-23-2009, 03:04 PM
Diakiao

I'm afraid I haven't gotten a response just yet. No biggie, just don't have a number to give ya.
07-24-2009, 03:05 PM
To-M

Engineers confirmed this ok and the messages you are receiving are not an issue. The reason is you are using asynchronous replication - which means it does not work all the time like synchronous mode, so this is part of the scheduling. Below are some details of the meanings that

Failed. Transient state following an I/O failure report by the local block device. Next state: Diskless.

Inconsistent. The data is inconsistent. This status occurs immediately upon creation of a new resource, on both nodes (before the initial full sync). Also, this status is found in one node (the synchronization target) during synchronization.

Outdated. Resource data is consistent, but outdated.

DUnknown. This state is used for the peer disk if no network connection is available.

Consistent. Consistent data of a node without connection. When the connection is established, it is decided whether the data are UpToDate or Outdated.

UpToDate. Consistent, up-to-date state of the data. This is the normal state.

A resource may have one of the following connection states:

StandAlone. No network configuration available. The resource has not yet been connected, or has been administratively disconnected (using drbdadm disconnect), or has dropped its connection due to failed authentication or split brain.

Disconnecting. Temporary state during disconnection. The next state is StandAlone.

Unconnected. Temporary state, prior to a connection attempt. Possible next states: WFConnection and WFReportParams.

Timeout. Temporary state following a timeout in the communication with the peer. Next state: Unconnected.

BrokenPipe. Temporary state after the connection to the peer was lost. Next state: Unconnected.

NetworkFailure. Temporary state after the connection to the partner was lost. Next state: Unconnected.

ProtocolError. Temporary state after the connection to the partner was lost. Next state: Unconnected.

TearDown. Temporary state. The peer is closing the connection. Next state: Unconnected.

WFConnection. This node is waiting until the peer node becomes visible on the network.

WFReportParams. TCP connection has been established, this node waits for the first network packet from the peer.

Connected. A DRBD connection has been established, data mirroring is now active. This is the normal state.

StartingSyncS. Full synchronization, initiated by the administrator, is just starting. The next possible states are: SyncSource or PausedSyncS.

StartingSyncT. Full synchronization, initiated by the administrator, is just starting. Next state: WFSyncUUID.

WFBitMapS. Partial synchronization is just starting. Next possible states: SyncSource or PausedSyncS.

WFBitMapT. Partial synchronization is just starting. Next possible state: WFSyncUUID.

WFSyncUUID. Synchronization is about to begin. Next possible states: SyncTarget or PausedSyncT.

SyncSource. Synchronization is currently running, with the local node being the source of synchronization.

SyncTarget. Synchronization is currently running, with the local node being the target of synchronization.

PausedSyncS. The local node is the source of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by drbdadm pause-sync.

PausedSyncT. The local node is the target of an ongoing synchronization, but synchronization is currently paused. This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by drbdadm pause-sync.

VerifyS. On-line device verification is currently running, with the local node being the source of verification.

VerifyT. On-line device verification is currently running, with the local node being the target of verification.
08-05-2009, 01:41 AM
tjk

Todd,

In async replication mode, how often does it check to see if it needs to send data to the replicated volume?

I setup an async mirror between 2 volumes, and I'm seeing some odd behavior, little to no network traffic between the nodes, yet I am making 2g to 4g new vm's and waiting for them to replicate over, seems like there is a delay, but I haven't seen anything in 10-15 minutes or more yet.

Protocol type:
Asynchronous
Connection:
StandAlone
Source info:
Logical volume:
lv0100
Consistency:
Consistent
Destination info:
Logical volume:
lv0100
Consistency:
n/a
IP address:
10.20.20.200

Says standalone, bewfore that it was wfbitmaps for connection. Any ideas?

Tom
08-06-2009, 02:49 PM
Diakiao

Much obliged!

Thanks for the info To-M! I have added that to our personal knowledge base. I don't remember seeing that info in the manual, though I might have missed it. If it isn't in there though I strongly recommend adding it! That resolved a lot of questions!

Thanks again!
08-09-2009, 07:45 PM
To-M

Quote:

Originally Posted by tjk

Says standalone, bewfore that it was wfbitmaps for connection. Any ideas?

The state is "standalone" means no replication is running at this moment.

The running async replication has no delay at all. When the source server has a write access the destination server is noted about this right away (different from synchronous replication).

If he want to have a delay you can set a schedule to this task but if the synchronization time will not be long enough after the task has stopped then the disk will be inconsistent.

We will try to write a "difference between sync and async replication" in KB in next week.
09-02-2009, 08:55 AM
lstcuser

Hi,

i have a synchronous replication task running which is used as a failover task. i also get those messages telling me that the volumes keep switching from consistent to inconsistent and from connected to synctarget.

as far as i understood now after reading this thread, this is just like an information that the replication/synchronization is running?!
if so, why do i need to get hundreds of e-mail notifications? firstly this is pretty annoying, when u get 20-30 mails per night, basicly telling u "hey, everything works allright". and secondly, i'm worried that if something will go wrong one day, i'll miss this one error mail among hundreds of its-ok-mails... can u somehow deactivate these non-error-mails?

thanks!
09-02-2009, 09:10 AM
lstcuser

forgot something:

why do i get these mail notifications only at certain times, especially at late evening and at night (when nobody is working)? why do i not get any of those mails during work time when there is some activity on the volume?
09-02-2009, 09:24 PM
To-M

There might be something else happening in the links, please send in the log file so we can review them.