Rsyncing Directories – Sanity Check

Home » CentOS » Rsyncing Directories – Sanity Check
CentOS 8 Comments

hey guys,

I need to mount a different volume onto /var so we have more room to breathe. I’ll be turning 3 servers into an elasticsearch cluster. And for some reason when the servers were ordered the large local volume ended up being /usr when the ES rpm likes to store it’s indexes on /var.

So I’m syncing the contents of both directories to a different place, and I’m going swap the large local volume from /usr to /var.

It looked like /opt had more than enough space to hold both directories.
/opt was 6GB and I successfully synced /usr to it. /usr was 2.5GB.

Then I went to sync /var to a temp folder in /opt. Checking I see that it still has 1/6GB available after the first sync.

# df -h /opt Filesystem Size Used *Avail* Use% Mounted on
/dev/mapper/SysVG-OptVol
6.0G 4.1G *1.6G* 72% /opt

The entire /var partition is only using 549MB:

# df -h /var Filesystem Size *Used* Avail Use% Mounted on
/dev/mapper/SysVG-VarVol
6.0G *549M* 5.1G 10% /var

So that being the case, if I make a temp directory in /opt called /opt/var, how come I am running out of space in doing my rsync? It fails at the end and the /opt volume is filled up to 100%. Even tho I only have 549MB to sync.

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
rsync: write failed on “/opt/var/log/lastlog”: No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6]
rsync: recv_generator: mkdir “/opt/var/www/manual/developer” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/opt/var/www/manual/faq” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/opt/var/www/manual/howto” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/opt/var/www/manual/images” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/opt/var/www/manual/misc” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir “/opt/var/www/manual/mod” failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: connection unexpectedly closed (148727 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[sender=3.0.6]

And if I do a df of the entire system, it looks like everything is still ok:

# df -h Filesystem Size Used Avail Use% Mounted on
/dev/mapper/SysVG-RootVol
2.0G 872M 1.1G 46% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 486M 87M 375M 19% /boot
/dev/mapper/SysVG-HomeVol
4.0G 137M 3.7G 4% /home
/dev/mapper/SysVG-OptVol
6.0G 4.3G 1.4G 76% /opt
/dev/mapper/SysVG-TmpVol
2.0G 130M 1.8G 7% /tmp
/dev/mapper/SysVG-UsrVol
197G 2.8G 185G 2% /usr
/dev/mapper/SysVG-VarVol
6.0G 549M 5.1G 10% /var

Does anyone have a good guess as to why these ‘out of space’ failures are occurring?

Thanks, Tim

8 thoughts on - Rsyncing Directories – Sanity Check

  • Hi Tim,

    At first glance, I don’t see anything obvious, but if it were me, I’d do the following:

    a) add the ‘n’ flag to do a dry run (no actual copying)

    b) increase rsync’s verbosity
    (A single -v will give you information about what files are being
    transferred and a brief summary at the end. Two -v options (-vv)
    will give you information on what files are being skipped and
    slightly more information at the end. A third ‘v’ is insanely
    verbose.)

    c) redirect standard out to a text file that you can examine for more
    clues.

    hth & regards,

    Carl

  • Are you running out of room for file and directory names, which is a different thing than simple free disk space?

  • Probaly sparse files or hard links? Try
    # rsync -aHASWXv –delete src/ dst/

  • Have you considered just resizing the volumes? If you’re trying to swap them with rsync, you’re going to have to reboot anyway, and relabel your system. If any daemons are running, you might also corrupt their data this way.

    Depending on what UIDs are allocated to your users, lastlog can be an enormous sparse file. You would need to use rsync’s -S flag to copy it.

  • Gordon Messmer wrote:

    Um, I’ve not been following this closely, but /var is 549M? And a separate partition? I haven’t had /var and /usr as separate partitions in almost 10
    years. Nor have I had a drive smaller than, um, 160G in about the same.

    That being said, why not simply mount the additional partition under var, for the directory that’s running out of space?

    mark

  • Hello Tim,

    lastlog is a VERY large SPARSE file and when you rsync it it looses the sparsity and tries to copy all the data to /opt

    ls -al -h /var/log/lastlog
    -rw-r–r–. 1 root root *94G* Jun 25 09:10 /var/log/lastlog

    Real space on disk
    du -h /var/log/lastlog
    *60K* /var/log/lastlog

    Lec

  • That’d probably be my preference. But in my role at this company I don’t have the direct access to do that. I’d probably have to open up a ticket to another department and have it done when ‘they get around to it’. In say 3
    or 4 weeks.