On 01/02/19 00:54, Ian Kent wrote:
On Thu, 2019-01-31 at 22:26 +0100, Frank Thommen wrote:
Dear all,
We have a weird issue with an automounter file, where a one-line change
leads to the situation, that after a while (10-30 minutes?, maybe less),
some of the mounts stop working on some(!) nodes (not always the same)
of our HPC cluster. This issue currently breaks our complete HPC
cluster. To make it worse, it is not really reproducible. However after
three days of checking back and forth we are desperate and hope to find
some helpful hint through the this maillist.
The symptom is, that trying to access one of the "virtual" automounted
directories is answered with "No such file or directory" (see below).
autofs doesn't seem to even try to mount the respective filesystem and I
cannot recognize any previous problem in the log (debugging mode).
Is it always the same directory that becomes unresponsive?
It's all the directories managed by this table.
Do you see anything at all in the log when accessing the directory
and getting the ENOENT?
nope. Absolutely nothing.
Does the problem also occur if you use a HUP signal to re-read the
maps?
Haven't tried this yet. We usually just restart autofs.
I have seen situations where an automount point becomes unresponsive
even though the kernel dentry flags appeared as they should to trigger
the automount.
I could never work out how this occurred but one possibility occurred
to me recently, see below.
We were initially mounting the following directory structure
(1)
/base
+--sub1
+-- sub11
| +-- [83 subdirectories via auto.sub11]
+-- sub12
+-- sub13
+-- sub14
+-- sub15
+-- sub16
| +-- sub161
+-- sub17
but needed to change this to
(2)
/base
+--sub1
| +-- sub11
| | +-- [83 subdirectories via auto.sub11]
| +-- sub12
| +-- sub13
| +-- sub14
| +-- sub15
| +-- sub16
| +-- sub161
+-- sub17
In /etc/auto.master we have:
--------------------------
/base /etc/auto.base browse
--------------------------
(there are others, but only the mounts configured through /etc/auto.base
are affected by the problem)
There are two variants of /etc/auto.base.
This variant works fine and doesn't seem to trigger any errors. It
represents the directory structure (1) which we needed to change :
--------------------------
sub1 /sub11 -fstype=autofs,vers=3,sec=sys file:/etc/auto.sub11
\
/sub12 -fstype=nfs,vers=3,sec=sys
share.big-fs1:/ifs/data/group/base/sub12 \
/sub13 -fstype=nfs,vers=3,sec=krb5
pool3.fast-fs1:/ifs/data/group1-sub13 \
/sub14 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub14 \
/sub15 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub15 \
/sub17 -fstype=nfs,vers=3,sec=sys
sub17.big-fs2:/ifs/sub17/data/sub17 \
/sub16/sub161 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub16
# /imaging file:/etc/auto.imaging
# sub12 is share.big-fs1:/ifs/data/group/base/sub12
--------------------------
This one results in autofs to be broken after a while on some nodes. It
represents the directory structure (2) (see above):
--------------------------
sub1 /sub11 -fstype=autofs,vers=3,sec=sys file:/etc/auto.sub11
\
/sub12 -fstype=nfs,vers=3,sec=sys
share.big-fs1:/ifs/data/group/base/sub12 \
/sub13 -fstype=nfs,vers=3,sec=krb5
pool3.fast-fs1:/ifs/data/group1-sub13 \
/sub14 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub14 \
/sub15 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub15 \
/sub16/sub161 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub16
sub17 -fstype=nfs,vers=3,sec=sys sub17.big-fs2:/ifs/sub17/data/sub17
--------------------------
Overnight I had this version of the file active:
--------------------------
sub1 /sub11 -fstype=autofs,vers=3,sec=sys file:/etc/auto.sub11
\
/sub12 -fstype=nfs,vers=3,sec=sys
share.big-fs1:/ifs/data/group/base/sub12 \
/sub13 -fstype=nfs,vers=3,sec=krb5
pool3.fast-fs1:/ifs/data/group1-sub13 \
/sub14 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub14 \
/sub15 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub15 \
/sub17 -fstype=nfs,vers=3,sec=sys sub17.big-fs2:/ifs/sub17 \
/sub16/sub161 -fstype=nfs,vers=3,sec=sys
share.big-fs2:/ifs/data/group/base/sub16
sub17 -fstype=nfs,vers=3,sec=sys sub17.big-fs2:/ifs/sub17/data/sub17
--------------------------
note that "sub17" appears twice in two locations. This version didn't
trigger the issue so far.
I fail to see any syntax problems in any of these two files variants.
The syntax looks ok to me too.
/etc/auto.sub11 is a list of 83 subdirectories for /base/sub1/sub11/.
No further mount options are defined in /etc/auto.sub11. It's a plain
"key server:share" list.
After deploying the "bad" file and restarting the automounter everything
works fine. After a while, problems start happening:
That delay could correspond to the expiring away of the submount
at sub11.
Recently RedHat QE reported a problem where directories could be
created in the autofs fs after automount shutdown when the autofs
fs remained mounted due to it being in use.
Nowadays it's important for the directory topology of the autofs
file system to remain unchanged over restart so the ability to
do this has been prevented in the kernel.
That kernel change won't be included in RHEL until 7.7.
The existence of a directory within sub11, perhaps created over
restart, might be preventing the sub11 submount from being mounted
again after the expire.
So it's probably better to use a HUP signal to re-read the maps
rather than a restart.
I will try that. However it doesn't really solve the issue ;-)
There's also the possibility that the sub11 mount has not been
properly expired away but that should show up in the log and as
long as the map entries relating to that map are the same automount
should reconstruct the mount tree ok at start up.
Yes, you need to be careful when using sub-mounts and modifying
your maps because the daemon doesn't know what the map was before
the restart and a previously valid directory might cause odd
problems.
I'm pretty sure, for the case of a submount being expired away
I don't unconditionally remove the entire directory tree, it's
done that way for historic reasons but could possibly be changed
now.
But, having said all this, I can't say for sure what's causing
this.
--------------------------
$ ls /base/sub1/
sub11 sub16 sub14 sub13 sub17 sub12 sub15
$ ls /base/sub1/sub11/
ls: cannot open directory /base/sub1/sub11/: No such file or directory
$
--------------------------
We found, that if we constantly run `\ls /base/sub1/sub11`, e.g. in a
loop, once per minute, then the problem doesn't seem to emerge.
We are running autofs 5.0.7, release 70.el7_4.1 on CentOS 7.4.1708.
Updating the CentOS release ist not possible due to hardware and
software constraints.
Is patching and re-building the CentOS kernel possible in your
environment?
Probably not.
There have been other problems related to the autofs directory
tree becoming out of sync leading to unresponsive mounts but
I don't think there was anything along those lines between 7.4
and now.
I can provide the debug log from a timepoint shortly after the
automounter has been restarted (everything ok) until a timepoint, when
the problem occurs.
It would probably be better to review this type of info. in a
bugzilla bug.
Problem is, that I cannot really say /when/ the problem started. 3000
lines of autofs debug log are probably not very usefyl ;-). Also I am
not sure if I would be allowed to give out logfiles to publicly readable
sites, as we have quite strict privacy and data protection regulations.
I'd rather not do that in CentOS bugzilla because if there is a
change needed in RHEL there needs to be a RHEL bug for me to get
it reviewed and possibly included.
You can always log a bug directly in RedHat bugzilla against the
corresponding RHEL release. It doesn't matter to me that your
using CentOS but ...
Be aware that bugs logged directly in Bugzilla are best effort
as time permits with bugs that come in via support or other
internal sources having priority, as you'd expect.
Ian
frank
