How to initially fill share using rsync

Hi!

Second try to post... :-(

Open-e DSS with up to 8 TB disk.
Successfully set up.
Successfully integrated in NIS environment.
Successfully created VG's.
Successfully created shares.
Successfully mounted the shares.

Set up "Data replication agent settings" for one of the shares.
Now I want to fill that share with some 100 GB of initial data.
I found that I may use rsync for that purpose.

But I fail:

- when using ssh, the connection is refused at port 22 -- no sshd on Open-e !?!

- when using rsh, after 10-15 seconds, I get this error:

Linux 2.6/rsync 2.6.8
=====================
# export RSYNC_PASSWORD=mypassword
# rsync -raRx --acls --xattrs --whole-file --delete --force --rsh=/path/to/rsh "./" myusername@shoe-2012:/share/mysharename/
shoe-2012: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(459) [sender=2.6.8]

Solaris 9/rsync 2.6.9 OR 3.0.4
==============================
# export RSYNC_PASSWORD=mypassword
# rsync -avrRulHpogt -n --delete --rsh=/path/to/rsh --force ./ 'myusername@shoe-2012:/share/mysharename/'
shoe-2012: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(632) [sender=3.0.4]


Any hint how to correct that?


Robert

Dear Robert,

Did you set the replication task?

Please, send your logs (Status->Hardware) to our support e-mail "support@open-e.com"

Dear Sh-J

Thank you for your answer.

Quote:

---

Originally Posted by Sh-J

Did you set the replication task?

---

No, I didn't. It's not clear to me why and how I have to create a data replication task when I want to import existing data from outside the Open-e-world to a share.

In the settings menu for these tasks, I found "source share", where I can only choose a share of the NAS. But I want to have the NAS share as the target... This menu looks like if it was to export data to another storage, not for import?!

Best,

Robert

Quote:

---

Originally Posted by the_nipper

Hi!
[...]
Set up "Data replication agent settings" for one of the shares.
Now I want to fill that share with some 100 GB of initial data.
I found that I may use rsync for that purpose.

But I fail:

- when using ssh, the connection is refused at port 22 -- no sshd on Open-e !?!

- when using rsh, after 10-15 seconds, I get this error:
[...]
Any hint how to correct that?

Robert

---

Robert, have you tried using rsync protocol instead of ssh or rsh? DSS provides an rsync daemon, which the client can connect to directly.

With regards,
Jens

Hi Jens,

yes, that's what I asked in my first posting: I used rsync, but I failed. Please look at my first posting where I described the commands that I used and the error messages that I got.

You or anyone else succeeded in using rsync? Then, please, tell me the client command line and the settings for Open-e...

Best,

Robert

Again, Jens:

Quote:

---

Originally Posted by jmo

Robert, have you tried using rsync protocol instead of ssh or rsh? DSS provides an rsync daemon, which the client can connect to directly.

---

Yes, I used rsync. But rsync is a protocol only. To use it, you need a connection. This connection may be done using rsh or ssh, e.g. depends on what the remote side allows.

I tried both ssh and rsh, without success.

Best,

Robert

Quote:

---

Originally Posted by the_nipper

Hi Jens,

yes, that's what I asked in my first posting: I used rsync, but I failed. Please look at my first posting where I described the commands that I used and the error messages that I got.

You or anyone else succeeded in using rsync? Then, please, tell me the client command line and the settings for Open-e...

Best,

Robert

---

Robert,

I'm sorry I wasn't specific enough: rsync knows two general modes of operation:
- using a remote shell to invoke rsync on the remote host and communicate via stdin+out
- using rsyncd on the remote host

DSS is providing the latter (rsyncd), your invocations were for the former (tying both ssh and rsh as the remote shell command).

Yes, I've been using rsync to both populate and to retrieve files via rsync to/from DSS.

With regards,
Jens

Quote:

---

Originally Posted by the_nipper

Again, Jens:

Yes, I used rsync. But rsync is a protocol only. To use it, you need a connection. This connection may be done using rsh or ssh, e.g. depends on what the remote side allows.

I tried both ssh and rsh, without success.

Best,

Robert

---

Robert,

I'm sorry to disagree with you limiting the connections to rsh and ssh - the following is an excerpt from the rsync command's man page:
--- cut here ---
GENERAL
[...]
There are two different ways for rsync to contact a remote system: using a remote-shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP. The remote-shell transport is used when‐ever the source or destination path contains a single colon (:) separator after a host specification. Contacting an rsync daemon directly happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified (see also the "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" section for an exception to this latter rule).
--- cut here ---

DSS only provides the latter - directly connecting to rsyncd via TCP. Neither rsh nor ssh are available on DSS.

With regards,
Jens

Hi Nipper

Try this as it has worked for other users:

You can use our NAS-R3 as a destination server for the rsync process. Enable the Data Replication Agent in WebGUI -> configuration -> NAS settings and also for share(s) that you want to have as destination in NAS Resource. Then set username and password if desired.

Rsync command used to synchronize from your operating system to Open-E server can look similar to this:

cd $SOURCE_DIR
export RSYNC_PASSWORD=${PASSWORD}
rsync -raRx --acls --xattrs --whole-file --delete --force "./"
"$LOGIN@$RSYNC_DEST_IP::$DEST_SHARE"

While using rsync without password please unset RSYNC_PASSWORD variable.

Recomend using a separate NIC for the rsync transfer.

You will need to export your UID and GID from your system then import them to the NAS-R3

For example in UID case - go to NAS resources -> Users -> Function: UID mappings (upload/download).

To upload UID:

* Find configuration file uid_mappings.csv (format:user_name;uid) by clicking "Browse" buuton. This file should be in UTF-8 encoding,
* Press "upload" button to import uid mappings,
* If there will be some errors while importing uids please read uid_mappings_import.log file logs package.

Also you have entered in your question to support please let them know tha

Hi To-L,

Quote:

---

Originally Posted by To-L

You will need to export your UID and GID from your system then import them to the NAS-R3

For example in UID case - go to NAS resources -> Users -> Function: UID mappings (upload/download).

---

I have to do this upload from my system as user root, e.g. UID/GID 0. I guess there's no mapping needed. I have to do this because the data are user homes which I only can access as the superuser.

All user UID/GID are well known to Open-e since it uses the same NIS service as my server.


Question: under these circumstances, is there really need of UID/GID mapping?


Robert/Nipper

Dear Robert,

Its good to map them, so your replicated volume can use/access the same groups and users.

How its going now?

Regards,
SJ

Hi!

Can't agree. Open-e is configured to use NIS. We have several hundreds of users, and we have daily changes on this database --really!!! (research environment close to a university)

Thus, Open-e MUST be able to work with NIS. It isn't a solution for us to keep out (actually) two Open-e systems manually consistent to NIS.

Best,

Robert

Hi all,

the problem is solved now. I didn't realize that rsync is used in _daemon-mode_, and therefore it needs double colon in the remote address. Furthermore, the NAS resource name must be given instead of a filesystem's path.

wrong:

rsync (options) ./ user@DSS:/share/resource

correct:

rync (options) ./ user@DSS::resource

Things are working well now. Thanks to all who helped!

Best,

Robert