4TB limitation on volume replication

[To-M]

We use protocol C - engineers found it to be faster than others.

Our kernel uses 10-20 MB of the Vmalloc memory. The replication Vmalloc usage is about Volume Size/32768.

[vkeven]

Wrong DRBD need 32MB of vmalloc by 1TB of storage, the formula is storage_in_GB/32 and not by 32768 so 4096GB/32 = 128 so I don't know if some people have a rock solid replication volume over 1TB but from the maker of DRBD 10 to 20MB is clearly not correct.

Read this post about memory allocation and 32MB by 1TB , the writer is the PROGRAMMER of DRBD Dipl-Ing Philipp Reisner so I don't know if your enginnier are reading this forum or even DRBD doc and mailing list but I think they should do , 32K by 1GB of storage so 1TB=1000GBx32Kb=32000Kb=32MBx4(for 4096GB or 4TB)=128MB,now I know why people complaining about replication stop working.

Just for Information Protocol C is probably the slowest one but the most secure for your data , so this is good , a control of protocol could be an interesting option , like protocol A for asynchronious remote site.

http://www.drbd.org/users-guide/s-re...protocols.html

[masim]

Hi...

Kevin - thanks for the nice info... but can you explain this:
I am replicating 2.8TB and getting an average steady speed of 62MB/s.

if what you are saying is right, then what I am getting is impossible...

Any one else can confirm this???

[To-M]

Here let me help Keven out as he wanted to place his trophy on our response (yes should have been better but without the blood).

Here is a better description and using Protocol C is better in the end. If you don’t like it build it yourself .

For each device, drbd will (try to) allocate X MB of bitmap, plus some constant amount (<1MB). X = storage_size_in_GB/32, so 1 TB storage -> 32 MB bitmap.

By default Linux allocates 128MB to Vmalloc. For systems using more than 4TB, this may cause an issue - but give us time as we resolve this!!! It will happen, we are not going to just update without hard testing with our main customers unlike what some want it now.

[To-M]

And yes asynchronous is in the works - very soon

[rykerlee]

We are replicating with 2 x 4TB volumes and haven’t had any issues for over 68 days, though we are using DSS ver. 5.0_42 in block io mode.

Not sure if this helps others but on a side note we would have more interest in what Heimic and SeanLeyne stated then the 16TB replication option (DRBD+). Vkeven must be needing something other then what DSS provides currently or wanting blood like Todd said

Todd - any chance that we will see asynchronous soon (not asynchronious )?

[To-M]

Long time no hear from U!

We want to say mid part of Q2 - though if there are issues or pending projects (especially from you guys for those huge orders ) then all this delays it.