Mount Bind Problem

Home » CentOS » Mount Bind Problem
CentOS 10 Comments

On CentOS release 6.7, there’s file system problem. Aftert reboot /home is set under /home (/home/home) like followings.

# ls -l /home total 48
-rw——- 1 root root 7168 Nov 15 19:10 aquota.group
-rw——- 1 root root 7168 Jan 25 11:09 aquota.user drwxr-x— 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 5 01:10 log drwx—— 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Dec 13 22:34 mysql

# df Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root
12778200 2946824 9175608 25% /
tmpfs 961176 0 961176 0% /dev/shm
/dev/vda1 487652 205564 256488 45% /boot
/dev/vdb 41153856 549492 38507212 2% /mnt/extradiskA
/dev/vdb 41153856 549492 38507212 2% /home

I have set on fstab /home with ‘mount bind’ but it seems like bind is not effective.

/etc/fstab is following.

# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul 28 08:21:01 2014
#
# Accessible filesystems, by reference, are maintained under ‘/dev/disk’
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root / ext4
defaults,usrquota,grpquota 1 1
UUID

10 thoughts on - Mount Bind Problem

  • —– Mail original —–

    Indeed. By default, nothing mounts “bind” filesystems. The init scripts don’t read the hole fstab file, there are filters to mount only some filesystems types. If you want to mount the “bind” FS on boot, you need to add the related commands into /etc/rc.d/rc.local.

    Sylvain. Pensez ENVIRONNEMENT : n’imprimer que si ncessaire

  • Nonsense! I’m also running CentOS 6.7 and have several bind-mounts in my /etc/fstab:

    /var/home /home none bind 0 0
    /var/lib/libvirt/etc /etc/libvirt none bind 0 0
    /srv/news /var/spool/news none bind 0 0

    All of them occur automatically when the system boots. The
    “mount -a” in /etc/rc.d/rc.sysinit takes care of them just fine. The filter on that command is _excluding_ certain types. Note that the first two letters are “no”, which applies to all the listed types:

    mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O
    no_netdev

    [rkn] ~ $ df Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/mapper/vg_omega3g-rootvol
    11963960 5995348 5337828 53% /
    tmpfs 8194160 340 8193820 1% /dev/shm
    /dev/sda1 689128 130200 508752 21% /boot
    /dev/mapper/vg_omega3g-var
    38166008 8282340 27921892 23% /var
    /dev/mapper/vg_omega3g-srvlv
    564310312 360619492 175002488 68% /srv
    /dev/mapper/vg_omega3g-virt
    194274580 119917252 64617656 65% /var/lib/libvirt tmpfs 2097152 652 2096500 1% /tmp
    [rkn] ~ $ df /home /etc/libvirt /var/spool/news Filesystem 1K-blocks Used Available Use% Mounted on
    /var/home 38166008 8282340 27921892 23% /home
    /var/lib/libvirt/etc 194274580 119917252 64617656 65% /etc/libvirt
    /srv/news 564310312 360619508 175002472 68% /var/spool/news

    —
    Bob Nichols “NOSPAM” is really part of my email address.
    Do NOT delete it.

  • In /etc/rc.d/rc.sysinit, there’s “mount -a -n -t nonfs,nfs4,smbfs,ncpfs,cifs,gfs,gfs2i,glusterfs -O
    no_netdev”.

    I have other same type servers. And on other servers with same /etc/fstsb, file system is correct. There’s no difference between normal server’s rc.sysinit and this problem server’s rc.sysinit too.

    Please tell me other possibility which I have to check.

    2016-04-12 23:58 GMT+09:00 Robert Nichols :

    CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS

  • It’s not apparent how /dev/vdb (the whole filesystem) came to be mounted on both /mnt/extradiskA and /home. Let’s see the output from “lsblk” and “mount”. Did you look in /mnt/extradiskA/home and verify that just the expected user directories are there
    (no “/mnt/extradiskA/home/home”)?

    It might also be worthwhile to verify that the mount point directory /mnt/extradiskA in the root filesystem is empty. If there happens to be a /mnt/extradiskA/home directory present before /dev/vdb gets mounted, that could allow the mounts to be performed out of order, though I don’t really see how that could lead the to result you have.

    mkdir /tmp/tmproot
    mount –bind / /tmp/tmproot
    ls -a /tmp/tmproot/mnt/extradiskA
    umount /tmp/tmproot

    —
    Bob Nichols “NOSPAM” is really part of my email address.
    Do NOT delete it.

  • Dear Robert,

    Thank you. The state after reboot are as follows.

    # LANG=C lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sr0 11:0 1 1024M 0 rom vda 252:0 0 15G 0 disk
    |-vda1 252:1 0 500M 0 part /boot
    `-vda2 252:2 0 14.5G 0 part
    |-VolGroup-lv_root (dm-0) 253:0 0 12.5G 0 lvm /
    `-VolGroup-lv_swap (dm-1) 253:1 0 2G 0 lvm [SWAP]
    vdb 252:16 0 40G 0 disk /mnt/extradiskA

    # mount
    /dev/mapper/VolGroup-lv_root on / type ext4 (rw,usrquota,grpquota)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw)
    /dev/vda1 on /boot type ext4 (rw)
    /dev/vdb on /mnt/extradiskA type ext4 (rw,usrquota,grpquota)
    /mnt/extradiskA/home on /home type none (rw,bind)
    /mnt/extradiskA/log on /var/log type none (rw,bind)
    /mnt/extradiskA/mysql on /var/lib/mysql type none (rw,bind)
    /mnt/extradiskA/.backup on /.backup type none (rw,bind)
    /mnt/extradiskA/.daily_backup on /.daily_backup type none (rw,bind)
    /mnt/extradiskA/backups on /var/backups type none (rw,bind)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    /dev/vdb on /home type ext4
    (rw,relatime,barrier=1,data=ordered,usrquota,grpquota)

    # ls -l /home total 48
    -rw——- 1 root root 7168 Nov 15 19:10 aquota.group
    -rw——- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-x— 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 14 10:23 log drwx—— 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql
    # ls -al /mnt/extradiskA
    total 72
    drwxr-xr-x 9 root root 4096 Nov 15 19:10 . drwxr-xr-x. 3 root root 4096 Nov 15 19:00 ..
    -rw——- 1 root root 7168 Nov 15 19:10 aquota.group
    -rw——- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-xr-x 2 root root 12288 Apr 14 00:00 .backup drwxr-x— 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 2 root root 4096 Nov 16 02:30 .daily_backup drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 14 10:23 log drwx—— 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql

    # ls -al /mnt/extradiskA/home total 16
    drwxr-xr-x 4 root root 4096 Jan 1 11:59 . drwxr-xr-x 9 root root 4096 Nov 15 19:10 .. drwxr-xr-x. 7 admin admin 4096 Sep 21 2015 admin drwxr-xr-x 4 kavfile kavfile 4096 Jan 1 12:00 kavfile

    # mkdir /tmp/tmproot
    # mount –bind / /tmp/tmproot
    # ls -a /tmp/tmproot/mnt/extradiskA
    . ..

    # cat /etc/fstab

    #
    # /etc/fstab
    # Created by anaconda on Mon Jul 28 08:21:01 2014
    #
    # Accessible filesystems, by reference, are maintained under ‘/dev/disk’
    # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
    #
    /dev/mapper/VolGroup-lv_root / ext4
    defaults,usrquota,grpquota 1 1
    UUID=9a855356-ee00-4ae7-a185-226566ce1dba /boot ext4
    defaults 1 2
    UUID=093c9b95-8336-4682-9927-132dbdf2f082 /mnt/extradiskA ext4
    defaults,usrquota,grpquota 1 2
    /dev/mapper/VolGroup-lv_swap swap swap defaults
    0 0
    tmpfs /dev/shm tmpfs defaults 0 0
    devpts /dev/pts devpts gid=5,mode=620 0 0
    sysfs /sys sysfs defaults 0 0
    proc /proc proc defaults 0 0
    /mnt/extradiskA/home /home none bind 0 0
    /mnt/extradiskA/log /var/log none bind 0 0
    /mnt/extradiskA/mysql /var/lib/mysql none bind 0 0
    /mnt/extradiskA/.backup /.backup none bind 0 0
    /mnt/extradiskA/.daily_backup /.daily_backup none bind 0 0
    /mnt/extradiskA/backups /var/backups none bind 0 0

    2016-04-13 23:32 GMT+09:00 Robert Nichols :

    CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS

  • ^^^^^^^^^^^^^^^^^^^^^^^^^^^

    Well, there it is, the extra mount of /dev/vdb on /home. It’s not apparent how it got that way. Since it appears to be at the bottom of /etc/mtab, it apparently happened _after_ all the other mounts. Some script must have done that. If it’s not in /etc/rc.d/rc.local, then I’d do a progressively wider search for references to /home, starting with

    grep -r /home /etc

    —
    Bob Nichols “NOSPAM” is really part of my email address.
    Do NOT delete it.

  • Dear Robert,

    Before sending ‘grep -r /home /etc’ data, I want tell you what happned this morning.

    In order to solve the /home/home problem, ‘umount /home’ had been done, system had been running in a normal file system. But suddenly /home has been lost.

    Key information of that time is as in the **** lines. And I found in the //// lines messages log. How do you think the reason of trouble.

    ********************
    # ls -l /home total 0
    # ls -l /
    total 3614
    -rw——- 1 root root 12288 Apr 15 05:00 aquota.group
    -rw——- 1 root root 10240 Apr 15 05:00 aquota.user dr-xr-xr-x. 2 root root 4096 Nov 15 18:35 bin dr-xr-xr-x. 5 root root 5120 Nov 15 18:36 boot drwxr-xr-x 2 root root 4096 Jul 24 2015 cgroup drwxr-xr-x 17 root root 3820 Apr 14 10:24 dev drwxr-xr-x. 75 root root 4096 Apr 14 10:31 etc drwxr-xr-x 2 root root 4096 Nov 15 19:04 home dr-xr-xr-x. 8 root root 4096 Aug 8 2015 lib dr-xr-xr-x. 8 root root 12288 Nov 15 18:35 lib64

    # ls -l /mnt/extradiskA
    total 48
    -rw——- 1 root root 7168 Apr 15 02:14 aquota.group
    -rw——- 1 root root 7168 Apr 14 10:22 aquota.user drwxr-x— 2 root root 4096 Jan 18 21:55 backups drwxr-xr-x 4 root root 4096 Jan 1 11:59 home drwxr-xr-x 8 root root 4096 Apr 15 01:10 log drwx—— 2 root root 16384 Nov 15 18:58 lost+found drwxr-xr-x 6 mysql mysql 4096 Apr 14 10:24 mysql
    # ls -l /mnt/extradiskA/home total 8
    drwxr-xr-x. 7 admin admin 4096 Sep 21 2015 admin drwxr-xr-x 4 kavfile kavfile 4096 Jan 1 12:00 kavfile
    # mount
    /dev/mapper/VolGroup-lv_root on / type ext4 (rw,usrquota,grpquota)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw)
    /dev/vda1 on /boot type ext4 (rw)
    /dev/vdb on /mnt/extradiskA type ext4 (rw,usrquota,grpquota)
    /mnt/extradiskA/home on /home type none (rw,bind)
    /mnt/extradiskA/log on /var/log type none (rw,bind)
    /mnt/extradiskA/mysql on /var/lib/mysql type none (rw,bind)
    /mnt/extradiskA/.backup on /.backup type none (rw,bind)
    /mnt/extradiskA/.daily_backup on /.daily_backup type none (rw,bind)
    /mnt/extradiskA/backups on /var/backups type none (rw,bind)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    / on /tmp/tmproot type none (rw,bind)
    # df Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/mapper/VolGroup-lv_root
    12778200 3149592 8972840 26% /
    tmpfs 961176 0 961176 0% /dev/shm
    /dev/vda1 487652 205564 256488 45% /boot
    /dev/vdb 41153856 542076 38514628 2% /mnt/extradiskA

    # cat /etc/fstab

    #
    # /etc/fstab
    # Created by anaconda on Mon Jul 28 08:21:01 2014
    #
    # Accessible filesystems, by reference, are maintained under ‘/dev/disk’
    # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
    #
    /dev/mapper/VolGroup-lv_root / ext4
    defaults,usrquota,grpquota 1 1
    UUID=9a855356-ee00-4ae7-a185-226566ce1dba /boot ext4
    defaults 1 2
    UUID=093c9b95-8336-4682-9927-132dbdf2f082 /mnt/extradiskA ext4
    defaults,usrquota,grpquota 1 2
    /dev/mapper/VolGroup-lv_swap swap swap defaults
    0 0
    tmpfs /dev/shm tmpfs defaults 0 0
    devpts /dev/pts devpts gid=5,mode=620 0 0
    sysfs /sys sysfs defaults 0 0
    proc /proc proc defaults 0 0
    /mnt/extradiskA/home /home none bind 0 0
    /mnt/extradiskA/log /var/log none bind 0 0
    /mnt/extradiskA/mysql /var/lib/mysql none bind 0 0
    /mnt/extradiskA/.backup /.backup none bind 0 0
    /mnt/extradiskA/.daily_backup /.daily_backup none bind 0 0
    /mnt/extradiskA/backups /var/backups none bind 0 0

    ********************

    ////////////////////////
    Apr 15 00:00:03 teisui2 kernel: ————[ cut here ]————
    Apr 15 00:00:03 teisui2 kernel: WARNING: at fs/ext4/inode.c:3945
    ext4_flush_unwritten_io+0x74/0x80 [ext4]() (Not tainted)
    Apr 15 00:00:03 teisui2 kernel: Hardware name: KVM
    Apr 15 00:00:03 teisui2 kernel: Modules linked in: dazukofs(U) ipv6
    xt_multiport iptable_filter ip_tables joydev sg virtio_balloon serio_raw virtio_net i2c_piix4 i2
    c_core ext4 jbd2 mbcache virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
    scsi_wait_scan]
    Apr 15 00:00:03 teisui2 kernel: Pid: 1707, comm: spamd Not tainted
    2.6.32-573.1.1.el6.x86_64 #1
    Apr 15 00:00:03 teisui2 kernel: Call Trace:
    Apr 15 00:00:03 teisui2 kernel: [] ?
    warn_slowpath_common+0x91/0xe0
    Apr 15 00:00:03 teisui2 kernel: [] ?
    warn_slowpath_null+0x1a/0x20
    Apr 15 00:00:03 teisui2 kernel: [] ?
    ext4_flush_unwritten_io+0x74/0x80 [ext4]
    Apr 15 00:00:03 teisui2 kernel: [] ?
    ext4_sync_file+0x88/0x1d0 [ext4]
    Apr 15 00:00:03 teisui2 kernel: [] ?
    dazukofs_unlocked_ioctl+0x10f/0x950 [dazukofs]
    Apr 15 00:00:03 teisui2 kernel: [] ?
    vfs_fsync_range+0xa1/0x100
    Apr 15 00:00:03 teisui2 kernel: [] ? vfs_fsync+0x1d/0x20
    Apr 15 00:00:03 teisui2 kernel: [] ? do_fsync+0x3e/0x60
    Apr 15 00:00:03 teisui2 kernel: [] ?
    sys_fdatasync+0x13/0x20
    Apr 15 00:00:03 teisui2 kernel: [] ?
    system_call_fastpath+0x16/0x1b Apr 15 00:00:03 teisui2 kernel: —[ end trace 6a7331f9bcd6af40 ]—
    ////////////////////////

    2016-04-14 23:18 GMT+09:00 Robert Nichols :

    CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS

  • Since you had two different things mounted on /home, it is unclear how the system would interpret “umount /home”. That is why I did not suggest doing that.

    From the error, it seems that some checks got bypassed and a busy filesystem was disconnected. It is likely that there is now some filesystem corruption on /dev/vdb, and you should reboot with a forced fsck to clean it up. You really should track down the cause of the extra mount of /dev/vdb on /home first, or the system will just come up wrong again.

    —
    Bob Nichols “NOSPAM” is really part of my email address.
    Do NOT delete it.