Cannot Open: No Space Left On Device

Home » CentOS » Cannot Open: No Space Left On Device
CentOS 9 Comments

I have a VBox VM running CentOS 6.7 and I want to extract the content of a
14G tar file. I’m doing as:

*tar -xvf www.tar.gz*

But I am getting a lot of errors like the ones shown below:

html/elclarinweb.dev/wp-content/uploads/2015/03/12-Aura-Ávila-400×320.jpg tar:
html/elclarinweb.dev/wp-content/uploads/2015/03/12-Aura-Ávila-400×320.jpg:
Cannot open: No space left on device html/elclarinweb.dev/wp-content/uploads/2015/03/Julianne-Moore-11-585×378.jpg tar:
html/elclarinweb.dev/wp-content/uploads/2015/03/Julianne-Moore-11-585×378.jpg:
Cannot open: No space left on device

So, trying to find why the VHDD (Virtual HDD) has left out of space I ran the following command:

*df -h*
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_server-lv_root 26G 24G 869M 97% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 477M 80M 373M 18% /boot

And then I’ve tried to find the conflictive files by running:

*du -hsx * | sort -rh | head -10*
du: cannot access `proc/3662/task/3662/fd/4′: No such file or directory du: cannot access `proc/3662/task/3662/fdinfo/4′: No such file or directory du: cannot access `proc/3662/fd/4′: No such file or directory du: cannot access `proc/3662/fdinfo/4′: No such file or directory
14G var
1.8G usr
278M lib
77M boot
31M etc
27M lib64
15M sbin
7.8M bin
188K dev
112K root

Why *df -h* is reporting 24G used? Where the space did go? How I can fix this?

9 thoughts on - Cannot Open: No Space Left On Device

  • Linux boxes typically reserve the last 5% of volume space for use by root only, so as far as your normal user is concerned, the volume is in fact full.

    You need to be careful with commands that descend into /proc like that, because its contents changes rapidly. Here, you’ve seen that PID 3662 disappeared between the time your * glob was evaluated and the du command actually tried to run on it.

    Your du -x argument says restrict to a single filesystem, but that is evaluated per FILE argument, so it doesn’t prevent du from walking off the LVM and into /proc. To do that, you’d need to say “du -hsx /”, but then you don’t get the results you want.

    Instead, I’d recommend just explicitly listing the most likely pigs: /usr, /var, /home, and /etc.

    I see about 16 GiB.

    24 – 16 = 8, which sounds suspiciously like the size of a swap file. What does mount say?

  • # du -sc /* /.??* –exclude /proc|sort -n
    0 /.autofsck
    0 /.autorelabel
    0 /misc
    0 /net
    0 /sys
    4 /cgroup
    4 /media
    4 /mnt
    4 /selinux
    4 /srv
    8 /opt
    16 /home
    16 /lost+found
    16 /tmp
    112 /root
    188 /dev
    7956 /bin
    14624 /sbin
    27088 /lib64
    31636 /etc
    78796 /boot
    284672 /lib
    1882548 /usr
    14422860 /var
    16750556 total

    lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sda 8:0 0 30G 0 disk
    ├─sda1 8:1 0 500M 0 part /boot
    └─sda2 8:2 0 29.5G 0 part
    ├─vg_server-lv_root (dm-0) 253:0 0 26.5G 0 lvm /
    └─vg_server-lv_swap (dm-1) 253:1 0 3G 0 lvm [SWAP]

  • did you (or someone else with root access) possibly delete a very large file in /var that may still have been in use? it’s very annoying but if you do a rm on a large file under /var that is still open by some process for writing, it won’t actually clear the space. you can overcome that by just truncating the file instead of doing an rm (e.g. either > /var/log/bigfile or cp /dev/null /var/log/bigfile). the only way I know to fix the problem once you’re having it is to force the process to close/reopen the file… either by killing & restarting or getting the process to do it if it’s got that designed in. in practice I’ve often found it much easier to reboot a machine to fix such a problem condition.

  • In fact, that’s exactly what I did. I have tried to open that 14G .tar.gz file from mc and it crashes several times so I delete the temporary files from /tmp and also from /home where originally I copied. I’ve rebooted the VM and now the output of df -h looks better:

    # df -h Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/vg_server-lv_root
    26G 16G 8.7G 65% /
    tmpfs 2.0G 0 2.0G 0% /dev/shm
    /dev/sda1 477M 80M 373M 18% /boot

  • This is a FAQ. For future reference, you can locate such files in Linux:

    ls -l /proc/*/fd | grep deleted

    Deleted files are still present in the filesystem until they are closed.

  • I mean run the “mount” command. Its output is what the program “says.”

    But never mind now, since you found the actual problem.

  • Ah, bogus command anyway. I thought swap space showed up in the mount table, but it doesn’t. swapon -s is a better command for showing what I wanted to know.