This a note for Future John: Using filesystem capabilities with rsync by Hazel Smith. Use the CAP_DAC_READ_SEARCH capability. AKA: how I learned to stop worrying and love CAP_DAC_READ_SEARCH. There are some more notes over on Linux Capabilities and rsync, from presentation to practice.
This great article Backing up data like the adult I supposedly am popped up on Lobste.rs today. I think I’m gonna get myself an rsync.net account.
// 2022-09-18 jj5 – UPDATE: I have had an rsync.net account for some time now, works great. <3
I was having an issue with my rsync command:
time rsync --progress --verbose --acls --xattrs --exclude="/lib/mysql/ibdata1" --exclude="*.tmp" --stats --human-readable --recursive --del --force --times --links --hard-links --executability --numeric-ids --owner --group --perms --sparse --compress-level=6 diligence-test:/usr/ /data/temp/2017-11-15-163925/usr/
getting stuck during “receiving incremental file list” then giving up with:
Timeout, server diligence-test not responding. rsync: connection unexpectedly closed (16384 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2] rsync: connection unexpectedly closed (32569 bytes received so far) [generator] rsync error: unexplained error (code 255) at io.c(235) [generator=3.1.2] Command exited with non-zero status 255
I could see the server-side process hang in select() with:
root@diligence-test:/home/jj5# ps aux | grep rsync
root 5421 0.0 0.1 15636 2660 ? Ss 18:55 0:00 rsync --server --sender -vlHogtpAXrSe.iLsfxC --numeric-ids . /usr/
root@diligence-test:/home/jj5# strace -p 5421
strace: Process 5421 attached
select(1, [0], [], [0], {42, 979828}) = 0 (Timeout)
Anyway I figured out how to dodge the problem by nominating –delete-before instead of –del, e.g.:
time rsync --progress --verbose --acls --xattrs --exclude="/lib/mysql/ibdata1" --exclude="*.tmp" --stats --human-readable --recursive --delete-before --force --times --links --hard-links --executability --numeric-ids --owner --group --perms --sparse --compress-level=6 diligence-test:/usr/ /data/temp/2017-11-15-163925/usr/
So I was getting a baffling “unexpected remote arg” error from rsync today. Eventually I figured out the problem was that my argument “–executability” had become “– executablility”, I think due to a copy and paste problem where I copied some shell script code from Vim in a Konsole terminal into another Vim in another Konsole terminal. Traps for young players! If you get baffled by this error try putting an ‘echo’ in front of the command and then resize your terminal window to see if that affects things…
So I was running a backup with rsync and I saw this:
lib/mysql/ibdata1
437.40M 33% 4.53MB/s 0:03:10
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.2]
rsync: connection unexpectedly closed (155602 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2]
The issue seems to be that if you’re using rsync compression and the remote file gets changed while the rsync copy is in progress then shit gets corrupted. My solution was to handle error level ’12’ and retry without compression. If the file changes while the rsync is in progress the file will be corrupt, so you shouldn’t rely on the integrity of such files.