Backup Of Large Filesystems >500GB

Home » CentOS » Backup Of Large Filesystems >500GB
CentOS 17 Comments

I’ve a small system that I use to support a number of churches. I
provide web and email for them. My current server is running CentOS 6.3
with paired 1TB drives in a RAID1 configuration. It works well. One filesystem is very large, >500GB, and contains numerous large files:
SQL, docs, church libraries in ebook and digital form, plus stored videos of church services.

My problem is that I’ve found no means of backing up that file system. Dump and tar both error out as exceeding the maximum size. Neither will backup just the video directory (the largest) even with compression.

Backup will be to an external (USB) removable HD.

Can any suggest a prog or a method to back up this filesystem?

mw

17 thoughts on - Backup Of Large Filesystems >500GB

  • Have you tried good old rsync? Or, if you want incremental backups, check rdiff-backup. I’m sure our list colleagues will come up with even more solutions.

  • People have already suggested rsync and rdiff-backup; there’s also rsnapshot which is built on top of rsync.

    Another option could be mdadm if your RAID1 is already an mdadm array. You can add your USB drive to the array, wait for it to rebuild, then remove it from the array. I’d be wary of backing up an SQL database in that way, but I’d be wary of using rsync, dump, or tar too, so be sure to take a real backup (e.g., mysqldump, pg_dump) of your database first.

    –keith

  • I’d take the extra step of reformatting the USB drive into an ext3
    filesystem, then just use rsync in a nightly cron job.

    We do it with about a dozen or so workstations here with user data either on internal disks, or other external USB drives. Works great.

  • I’ve used rsync for remote file xfer of directory trees. It’s been awhile, I’d forgotten about it.

    mw

  • I use rsync to make a daily back up of my data on a second drive that is not normally mounted except when the backup is running. The drive is inside the same box so this backup is still subject to loss if the box is stolen or destroyed. I really should be backing up to an off site location.

    If you have a fire, flood, or other general disaster your local backup on an external drive isn’t going to buy you anything unless you store the external drive off site.


    _
    °v°
    /(_)\
    ^ ^ Mark LaPierre Registered Linux user No #267004
    https://linuxcounter.net/
    ****

  • FAT32 can go to 2TB (you just can’t format one that size in windows), but has the 4GB file size limit.

  • Unless you have a desperate need for this disk to be readable by Windows, I would definitely make an ext3, ext4, or xfs filesystem on that USB drive. It is absolutely not worth the headache you’ll have trying to restore properly from NTFS or FAT??.

    –keith

  • Reformatting is the obvious solution so you can use one of the rsync-based backups – but if you really had to keep the FAT format you could use tar |split -b with some reasonable size for output.

  • +1 for reformatting and using a file system that supports large(r) files

    Although not recommended…
    … you could create a file with dd (that will take some time) on the NTFS
    drive, and then loopback mount it and format it ext3 (or whatever you choose). Then mount that and go on your merry way with your rsync backups.

    *Disclaimer*: This solution is ugly and I’m almost certainly I will get tomatoes thrown at me for suggesting it. ;) Which includes converting FAT32 to NTFS if you don’t already have NTFS on the drive.

    There are more gotchas to what I suggested than just reformatting the drive ext3, etc and then sharing it out via Samba (or NFS) to your Windows clients (if you need to).

  • Or, you can use http://sourceforge.net/projects/ext2read/ (or similar)

    Copy&Paste:

    Ext2Read is an explorer like utility to explore ext2/ext3/ext4 files. It now supports LVM2 and EXT4 extents. It can be used to view and copy files and folders. It can recursively copy entire folders. It can also be used to view and copy disk and file

  • You can place that HDD in a rack (eSATA?) and add another that is kept off-site, and replace/swap them every week?

    Full off-site backup would require another box off-site with internet/wireless link for backups. Since you use rsync, data transfer should be fairly minimal.

  • note that SQL database files generally can’t be backed up safely while the SQL database server is active. either take a ‘dump’ or whatever backup of the database, and backup that dump rather than the raw SQL
    files, or stop the SQL service before making the backup, then restart it afterwards. Details vary per SQL server, of course.

    what would work really well for your general backup requirements, ignoring the above issue, is BackupPC. build an onsite BackupPC server with a file system sufficiently large to hold all backups you want kept online (I generally do 2 weeks of incrementals), and setup BackupPC
    archiving to move copies of completed backups to an offsite repository for disaster recovery.

LEAVE A COMMENT