Fastest Way Of Removing Very Large Number Of Files?

Home » CentOS » Fastest Way Of Removing Very Large Number Of Files?
CentOS 6 Comments

Hi All.

I currently have a problem on my backup server with very large number of small files in a large number of directories. I would like to delete them as fast as possible. Currently I use:

rsync -a –delete /empty_directory/ dir_to_clean/

I’ve read that rsync will be faster than rm or find. Can you recommend something? I use an ext4 filesystem.

Best regards, Rafal.

6 thoughts on - Fastest Way Of Removing Very Large Number Of Files?

  • I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements.

    Regards,

    Xinyun ZHOU

  • —– Original Message —–
    | Hi All.
    |
    | I currently have a problem on my backup server with very large number
    | of
    | small files in a large number of directories. I would like to delete
    | them
    | as fast as possible. Currently I use:
    |
    | rsync -a –delete /empty_directory/ dir_to_clean/
    |
    | I’ve read that rsync will be faster than rm or find.
    | Can you recommend something? I use an ext4 filesystem.
    |
    | Best regards,
    | Rafal.

    If it’s on a different file system than mkfs is the fastest. ;)


    James A. Peltier Manager, IT Services – Research Computing Group Simon Fraser University – Burnaby Campus Phone : 778-782-6573
    Fax : 778-782-3045
    E-Mail : jpeltier@sfu.ca Website : http://www.sfu.ca/itservices

    “A successful person is one who can lay a solid foundation from the bricks others have thrown at them.” -David Brinkley via Luke Shaw

  • I doubt saving functions calls is going to gain you anything in this case as 99.9% of the time the rm takes is on disk I/O. If you want to reduce the rm time you have to find a way to reduce the disk I/O it requires.

    Regards,
    Dennis

  • Correct.

    If it’s a whole directory (tree) that needs removing then I find

    mv dir dir.o ; mkdir dir ; chown ##:## dir; chmod ### dir ; rm -r dir.o &

    type stuff works just fine; the rm can chunk along in the background while there’s now a nice clean empty directory for the application.

  • In article <20130823104855.GA15299@mercury.spuddy.org>, Stephen Harris wrote:

    You can even dispense with the ## variable parts. And I would get the owner and modes correct before making the new directory appear:

    mkdir dir.n && chown –reference=dir dir.n && chmod –reference=dir dir.n && \
    mv dir dir.o && mv dir.n dir && echo rm -fr dir.o \; echo dir.o deleted | batch

    That will do the removal in a batch job without hanging on to the tty, and will email you a quick note (the output from batch) when it’s done.

    Cheers Tony

LEAVE A COMMENT