Odd Inconsistency With NFS

Home » CentOS » Odd Inconsistency With NFS
CentOS 4 Comments

I’m having an interesting/odd problem with nfs (I think). We recently (Monday/Tuesday) upgraded our file server from an ancient redhat 7.3 system to a shiny new CentOS 6.4 system. We don’t see any issues between the other CentOS boxes, but things get a bit weird when we start mounting on the old solaris clients.

The initial symptom was that the ‘tab complete‘ wasn’t working, and then we noticed that typing ‘ls *’ in the mounted directory was bombing. I tried forcing the mounting back to nfs3 but it’s not consistent. I’ve set up two boxes as servers and one of the solaris boxes is my client. Each server has two shares that are mounted on the client. Of those four, one of them works properly and the other three do not. I’ve spent most of the day trying to debug this and I cannot for the life of me tell why one share works and the rest don’t. Nothing seems to be special about that share versus the rest.

Here are notes on how things are set up.

on duke: (nis server)
vi /etc/ypfiles/automap
scrs1_bolt -soft,intr,retrans=1 boltzmann:/scrs1_bolt
summit_bolt -soft,intr,retrans=1 boltzmann:/summit_bolt
scrs1.mirror -soft,intr,retrans=1 goblin:/scrs1.mirror
summit.mirror -soft,intr,retrans=1 goblin:/summit.mirror
( cd /var/yp ; make )

on boltzmann: (nfs server)
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 50G 13G 37G 26% /
tmpfs 3.9G 1.2M 3.9G 1% /dev/shm
/dev/sdb3 177G 188M 175G 1% /aux
/dev/sda3 208G 44G 164G 21% /aux2
mkdir /aux/scrs1_bolt
mkdir /aux2/summit_bolt
ln -s /aux/scrs1_bolt /scrs1_bolt
ln -s /aux2/summit_bolt /summit_bolt
chmod 777 /aux/scrs1_bolt /aux2/summit_bolt
service nfs restart
vi /etc/exports
/scrs1_bolt xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
/summit_bolt xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
exportfs -rv

on bigdog: (client)
mkdir /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2
touch /tmp/test/b1/nothing_is_mounted
/tmp/test/b2/nothing_is_mounted /tmp/test/g1/nothing_is_mounted
/tmp/test/g2/nothing_is_mounted
mount -F nfs -o nfsvers=3 boltzmann:/summit_bolt /tmp/test/b1
mount -F nfs -o nfsvers=3 boltzmann:/scrs1_bolt /tmp/test/b2
mount -F nfs -o nfsvers=3 goblin:/summit.mirror /tmp/test/g1
mount -F nfs -o nfsvers=3 goblin:/scrs1.mirror /tmp/test/g2
ls -l /tmp/test/*
-rw-r–r– 1 root other 0 Jul 3 14:39
/tmp/test/nothing_is_mounted

/tmp/test/b1:
total 24
-rw-r–r– 1 root other 0 Jul 3 12:32 SUMMIT_BOLT
-rw-r–r– 1 root other 0 Jul 3 09:26
boltzmann_test_summit

/tmp/test/b2:
total 32
-rw-r–r– 1 root other 0 Jul 3 12:31 SCRS1_BOLT
-rw-r–r– 1 root other 0 Jul 3 09:26
boltzmann_test_scrs1

/tmp/test/g1:
total 280
-rw-r–r– 1 root other 0 Jul 3 15:40
.00_summit_nas_volume
-rw-rw-r– 1 root other 0 Jul 3 15:03
SUMMIT_MIRROR

/tmp/test/g2:
total 120
-rw-r–r– 1 root other 0 Jul 3 15:40
.00_scrs1_nas_volume
-rw-rw-r– 1 root other 0 Jul 3 15:02 SCRS1_MIRROR

ls -la /tmp/test/b1/*
-rw-r–r– 1 root other 0 Jul 3 12:32
/tmp/test/b1/SUMMIT_BOLT
-rw-r–r– 1 root other 0 Jul 3 09:26
/tmp/test/b1/boltzmann_test_summit

ls -l /tmp/test/b2/*
ls: No match.

ls -l /tmp/test/g1/*
ls: No match.

ls -l /tmp/test/g2/*
ls: No match.

mount
/tmp/test/b1 on boltzmann:/summit_bolt read/write/remote on Wed Jul 3 15:41:11 2013
/tmp/test/b2 on boltzmann:/scrs1_bolt read/write/remote on Wed Jul 3 15:41:11 2013
/tmp/test/g1 on goblin:/summit.mirror read/write/remote on Wed Jul 3 15:41:11 2013
/tmp/test/g2 on goblin:/scrs1.mirror read/write/remote on Wed Jul 3 15:41:11 2013

umount -a /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2

If I use automount to access the shares on the client, the mounts are made by default with nfs4 and we see this same ‘no match’ behaviour. It’s probably something really stupid but I’m just not seeing it…

If anyone has ideas and/or needs more info, please let me know.

4 thoughts on - Odd Inconsistency With NFS

  • Step 1 in debugging and troubleshooting… Use the KISS principle.

    Right now in that you have NIS, NFS, CentOS server and Solaris client
    (version? Given the red hat 7.3 instance you had would a safe assumption be not 11 or even 10?)

    Cut this down to work out which cog in the wheel is broken as there are just too many variables…

    Can you mount the NFS share locally to another directory on the NFS server?
    If that works can you mount it with a CentOS 6.4 client on another system?

    It’s been a while since I had to deal with NFS but I see you have symlinks in exports… I thought at least under 3 that was definitely not supported and you should use bind mounts…

    http://mail-index.netbsd.org/tech-kern/1995/05/28/0000.html

    Start there and see how you go…

  • —– Original Message —–
    | I’m having an interesting/odd problem with nfs (I think). We recently
    | (Monday/Tuesday) upgraded our file server from an ancient redhat 7.3
    | system to a shiny new CentOS 6.4 system. We don’t see any issues
    | between
    | the other CentOS boxes, but things get a bit weird when we start
    | mounting on the old solaris clients.
    |
    | The initial symptom was that the ‘tab complete’ wasn’t working, and
    | then
    | we noticed that typing ‘ls *’ in the mounted directory was bombing. I
    | tried forcing the mounting back to nfs3 but it’s not consistent. I’ve
    | set up two boxes as servers and one of the solaris boxes is my
    | client.
    | Each server has two shares that are mounted on the client. Of those
    | four, one of them works properly and the other three do not. I’ve
    | spent
    | most of the day trying to debug this and I cannot for the life of me
    | tell why one share works and the rest don’t. Nothing seems to be
    | special
    | about that share versus the rest.
    |
    | Here are notes on how things are set up.
    |
    | on duke: (nis server)
    | vi /etc/ypfiles/automap
    | scrs1_bolt -soft,intr,retrans=1 boltzmann:/scrs1_bolt
    | summit_bolt -soft,intr,retrans=1 boltzmann:/summit_bolt
    | scrs1.mirror -soft,intr,retrans=1 goblin:/scrs1.mirror
    | summit.mirror -soft,intr,retrans=1 goblin:/summit.mirror
    | ( cd /var/yp ; make )
    |
    | on boltzmann: (nfs server)
    | df -h
    | Filesystem Size Used Avail Use% Mounted on
    | /dev/sdb2 50G 13G 37G 26% /
    | tmpfs 3.9G 1.2M 3.9G 1% /dev/shm
    | /dev/sdb3 177G 188M 175G 1% /aux
    | /dev/sda3 208G 44G 164G 21% /aux2
    | mkdir /aux/scrs1_bolt
    | mkdir /aux2/summit_bolt
    | ln -s /aux/scrs1_bolt /scrs1_bolt
    | ln -s /aux2/summit_bolt /summit_bolt
    | chmod 777 /aux/scrs1_bolt /aux2/summit_bolt
    | service nfs restart
    | vi /etc/exports
    | /scrs1_bolt
    | xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
    | /summit_bolt
    | xxx.xxx.xxx.0/24(rw,no_root_squash,sync,insecure)
    | exportfs -rv
    |
    | on bigdog: (client)
    | mkdir /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2
    | touch /tmp/test/b1/nothing_is_mounted
    | /tmp/test/b2/nothing_is_mounted /tmp/test/g1/nothing_is_mounted
    | /tmp/test/g2/nothing_is_mounted
    | mount -F nfs -o nfsvers=3 boltzmann:/summit_bolt /tmp/test/b1
    | mount -F nfs -o nfsvers=3 boltzmann:/scrs1_bolt /tmp/test/b2
    | mount -F nfs -o nfsvers=3 goblin:/summit.mirror /tmp/test/g1
    | mount -F nfs -o nfsvers=3 goblin:/scrs1.mirror /tmp/test/g2
    | ls -l /tmp/test/*
    | -rw-r–r– 1 root other 0 Jul 3 14:39
    | /tmp/test/nothing_is_mounted
    |
    | /tmp/test/b1:
    | total 24
    | -rw-r–r– 1 root other 0 Jul 3 12:32
    | SUMMIT_BOLT
    | -rw-r–r– 1 root other 0 Jul 3 09:26
    | boltzmann_test_summit
    |
    | /tmp/test/b2:
    | total 32
    | -rw-r–r– 1 root other 0 Jul 3 12:31
    | SCRS1_BOLT
    | -rw-r–r– 1 root other 0 Jul 3 09:26
    | boltzmann_test_scrs1
    |
    | /tmp/test/g1:
    | total 280
    | -rw-r–r– 1 root other 0 Jul 3 15:40
    | .00_summit_nas_volume
    | -rw-rw-r– 1 root other 0 Jul 3 15:03
    | SUMMIT_MIRROR
    |
    | /tmp/test/g2:
    | total 120
    | -rw-r–r– 1 root other 0 Jul 3 15:40
    | .00_scrs1_nas_volume
    | -rw-rw-r– 1 root other 0 Jul 3 15:02
    | SCRS1_MIRROR
    |
    | ls -la /tmp/test/b1/*
    | -rw-r–r– 1 root other 0 Jul 3 12:32
    | /tmp/test/b1/SUMMIT_BOLT
    | -rw-r–r– 1 root other 0 Jul 3 09:26
    | /tmp/test/b1/boltzmann_test_summit
    |
    | ls -l /tmp/test/b2/*
    | ls: No match.
    |
    | ls -l /tmp/test/g1/*
    | ls: No match.
    |
    | ls -l /tmp/test/g2/*
    | ls: No match.
    |
    | mount
    | /tmp/test/b1 on boltzmann:/summit_bolt read/write/remote on
    | Wed Jul 3 15:41:11 2013
    | /tmp/test/b2 on boltzmann:/scrs1_bolt read/write/remote on
    | Wed
    | Jul 3 15:41:11 2013
    | /tmp/test/g1 on goblin:/summit.mirror read/write/remote on
    | Wed
    | Jul 3 15:41:11 2013
    | /tmp/test/g2 on goblin:/scrs1.mirror read/write/remote on
    | Wed
    | Jul 3 15:41:11 2013
    |
    | umount -a /tmp/test/b1 /tmp/test/b2 /tmp/test/g1 /tmp/test/g2
    |
    | If I use automount to access the shares on the client, the mounts are
    | made by default with nfs4 and we see this same ‘no match’ behaviour.
    | It’s probably something really stupid but I’m just not seeing it…
    |
    | If anyone has ideas and/or needs more info, please let me know.
    |
    | —
    | Thanks!
    | Miranda

    in our automounts we specified vers=3 as an option which consistently mounted with NFSv3. You may also wish to check out /etc/sysconfig/nfs and see if there are any inconsistencies there between the working and non-working servers. Note, you can specify the NFS version there too ;)


    James A. Peltier Manager, IT Services – Research Computing Group Simon Fraser University – Burnaby Campus Phone : 778-782-6573
    Fax : 778-782-3045
    E-Mail : jpeltier@sfu.ca Website : http://www.sfu.ca/itservices

    “A successful person is one who can lay a solid foundation from the bricks others have thrown at them.” -David Brinkley via Luke Shaw

  • James H, James P, and Leon, thanks for your replies. Here’s what I have in answer to your questions/suggestions:

    We do not use tcp_wrappers or iptables. The functional server is the redhat 7.3 box, the semi-functional server is the CentOS 6.4 box. On the CentOS 6.4 box, one share works as expected and the other does not. The use of symlinks does not appear to matter as both shares use them and one works while the other does not.

    For now, I am ignoring automounting and mounting everything manually so that I can explicitly declare the options I want each time.

    A summary of the testing (detailed output below):

    The two shares on the CentOS 6.4 server have been mounted on a CentOS
    6.4 client, a solaris 5.8 client, and a solaris 5.6 client. The CentOS
    6.4 client and solaris 5.8 client are able to successfully execute an
    ‘ls *’ on both the mounted shares, the solaris 5.6 client is not able to successfully execute ‘ls *’ on one share but is able to do so on the other. The two shares are each sub-directories of two partitions mounted on the CentOS 6.4 server. As far as I can tell, the two partitions are set up the same way, only difference is the naming. Other than the
    /etc/exports file, no other nfs-related configuration has been done on the server so the config files are the vanilla versions created when the packages are installed.

    I would like to be able to have the solaris 5.6 client be able to mount a given share and operate in the same manner as the rest of the clients, execute ‘ls *’ and whatever else. My coworker who is the one that brought this to my attention suggested that the issue might be with nfs versions, since the old server used nfs3 and the new one can use nfs4, but I’ve explicitly forced nfs3 on all the mounts and it doesn’t appear to make a difference.

    Thanks!
    Miranda

    —————————————————

LEAVE A COMMENT