On Thu, 2019-01-31 at 22:26 +0100, Frank Thommen wrote: > Dear all, > > We have a weird issue with an automounter file, where a one-line change > leads to the situation, that after a while (10-30 minutes?, maybe less), > some of the mounts stop working on some(!) nodes (not always the same) > of our HPC cluster. This issue currently breaks our complete HPC > cluster. To make it worse, it is not really reproducible. However after > three days of checking back and forth we are desperate and hope to find > some helpful hint through the this maillist. > > The symptom is, that trying to access one of the "virtual" automounted > directories is answered with "No such file or directory" (see below). > autofs doesn't seem to even try to mount the respective filesystem and I > cannot recognize any previous problem in the log (debugging mode). Is it always the same directory that becomes unresponsive? Do you see anything at all in the log when accessing the directory and getting the ENOENT? Does the problem also occur if you use a HUP signal to re-read the maps? I have seen situations where an automount point becomes unresponsive even though the kernel dentry flags appeared as they should to trigger the automount. I could never work out how this occurred but one possibility occurred to me recently, see below. > > We were initially mounting the following directory structure > > (1) > /base > +--sub1 > +-- sub11 > | +-- [83 subdirectories via auto.sub11] > +-- sub12 > +-- sub13 > +-- sub14 > +-- sub15 > +-- sub16 > | +-- sub161 > +-- sub17 > > > but needed to change this to > > (2) > /base > +--sub1 > | +-- sub11 > | | +-- [83 subdirectories via auto.sub11] > | +-- sub12 > | +-- sub13 > | +-- sub14 > | +-- sub15 > | +-- sub16 > | +-- sub161 > +-- sub17 > > > In /etc/auto.master we have: > > -------------------------- > /base /etc/auto.base browse > -------------------------- > > (there are others, but only the mounts configured through /etc/auto.base > are affected by the problem) > > > There are two variants of /etc/auto.base. > > This variant works fine and doesn't seem to trigger any errors. It > represents the directory structure (1) which we needed to change : > -------------------------- > sub1 /sub11 -fstype=autofs,vers=3,sec=sys file:/etc/auto.sub11 > \ > /sub12 -fstype=nfs,vers=3,sec=sys > share.big-fs1:/ifs/data/group/base/sub12 \ > /sub13 -fstype=nfs,vers=3,sec=krb5 > pool3.fast-fs1:/ifs/data/group1-sub13 \ > /sub14 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub14 \ > /sub15 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub15 \ > /sub17 -fstype=nfs,vers=3,sec=sys > sub17.big-fs2:/ifs/sub17/data/sub17 \ > /sub16/sub161 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub16 > # /imaging file:/etc/auto.imaging > # sub12 is share.big-fs1:/ifs/data/group/base/sub12 > -------------------------- > > > This one results in autofs to be broken after a while on some nodes. It > represents the directory structure (2) (see above): > -------------------------- > sub1 /sub11 -fstype=autofs,vers=3,sec=sys file:/etc/auto.sub11 > \ > /sub12 -fstype=nfs,vers=3,sec=sys > share.big-fs1:/ifs/data/group/base/sub12 \ > /sub13 -fstype=nfs,vers=3,sec=krb5 > pool3.fast-fs1:/ifs/data/group1-sub13 \ > /sub14 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub14 \ > /sub15 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub15 \ > /sub16/sub161 -fstype=nfs,vers=3,sec=sys > share.big-fs2:/ifs/data/group/base/sub16 > > sub17 -fstype=nfs,vers=3,sec=sys sub17.big-fs2:/ifs/sub17/data/sub17 > -------------------------- > > I fail to see any syntax problems in any of these two files variants. The syntax looks ok to me too. > > /etc/auto.sub11 is a list of 83 subdirectories for /base/sub1/sub11/. > No further mount options are defined in /etc/auto.sub11. It's a plain > "key server:share" list. > > After deploying the "bad" file and restarting the automounter everything > works fine. After a while, problems start happening: That delay could correspond to the expiring away of the submount at sub11. Recently RedHat QE reported a problem where directories could be created in the autofs fs after automount shutdown when the autofs fs remained mounted due to it being in use. Nowadays it's important for the directory topology of the autofs file system to remain unchanged over restart so the ability to do this has been prevented in the kernel. That kernel change won't be included in RHEL until 7.7. The existence of a directory within sub11, perhaps created over restart, might be preventing the sub11 submount from being mounted again after the expire. So it's probably better to use a HUP signal to re-read the maps rather than a restart. There's also the possibility that the sub11 mount has not been properly expired away but that should show up in the log and as long as the map entries relating to that map are the same automount should reconstruct the mount tree ok at start up. Yes, you need to be careful when using sub-mounts and modifying your maps because the daemon doesn't know what the map was before the restart and a previously valid directory might cause odd problems. I'm pretty sure, for the case of a submount being expired away I don't unconditionally remove the entire directory tree, it's done that way for historic reasons but could possibly be changed now. But, having said all this, I can't say for sure what's causing this. > > -------------------------- > $ ls /base/sub1/ > sub11 sub16 sub14 sub13 sub17 sub12 sub15 > $ ls /base/sub1/sub11/ > ls: cannot open directory /base/sub1/sub11/: No such file or directory > $ > -------------------------- > > We found, that if we constantly run `\ls /base/sub1/sub11`, e.g. in a > loop, once per minute, then the problem doesn't seem to emerge. > > We are running autofs 5.0.7, release 70.el7_4.1 on CentOS 7.4.1708. > Updating the CentOS release ist not possible due to hardware and > software constraints. Is patching and re-building the CentOS kernel possible in your environment? There have been other problems related to the autofs directory tree becoming out of sync leading to unresponsive mounts but I don't think there was anything along those lines between 7.4 and now. > > > I can provide the debug log from a timepoint shortly after the > automounter has been restarted (everything ok) until a timepoint, when > the problem occurs. It would probably be better to review this type of info. in a bugzilla bug. I'd rather not do that in CentOS bugzilla because if there is a change needed in RHEL there needs to be a RHEL bug for me to get it reviewed and possibly included. You can always log a bug directly in RedHat bugzilla against the corresponding RHEL release. It doesn't matter to me that your using CentOS but ... Be aware that bugs logged directly in Bugzilla are best effort as time permits with bugs that come in via support or other internal sources having priority, as you'd expect. Ian
