On 12/6/24 02:40, Nathaniel Middleton wrote:
Summary:
We define a /mount/path/pathcontd/mountpoint tree in autofs, have historically --ghosted the paths, but have recently attempted to set the offset mounts as browse-able.
Our configuration works great... at first. Once the timeout hits (I set this to 10s for debugging... but normally have it set to 300s), the autofs (type) mount points start expiring. Once this expiration starts, the offset mounts disappear (often leaving one mount point remaining that is broken). The disappeared mount points are unable to be re-mounted. This has hit us at semi-regular intervals on Centos 7 , but Rocky 8 servers are highly repro-able. This could be due to the lack of activity on the new systems though.
Right, the description sounds like there's been a partial expire of the
multi-mount (or possibly only
a directory has not been removed).
There are a couple of things to be aware of, first these multi-mounts
will create the sub-directory tree
below them and remove it when thet expire, that's normal behviour. Only
the root of the map entry is affected
by the browse setting.
Config:
distro Name: Rocky Linux 8.8 (Green Obsidian)
autofs-5.1.4-102.el8_8.2
NFS4.{0..2}(though... the nfs mounts are not being invoked when this issue happens.)
Autofs configs:
Auto.master:
/allen/programs/celltypes /etc/auto.programs_celltypes tcp hard intr timeo=600 retrans=2 async --ghost
auto.programs_celltypes(snipping out similar lines to conserve space):
production -fstype=nfs4 /incoming aidc-isi1-prd:/ifs/incoming/aibs/celltypes \
/0378 aidc-isi1-prd:/ifs/programs/celltypes/production/0378 \
/brainstim aidc-isi1-prd:/ifs/programs/celltypes/production/brainstim \
/conn aidc-isi1-prd:/ifs/programs/celltypes/production/conn
workgroups /0285 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0285 \
/0378 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0378
We have a couple similar mounts in a similar mount point to "/allen/programs/<different name>/<mount>" if it makes a difference there.
What this looks like:
Temporarily working:
ls /allen/programs/celltypes/production
incoming 0378 brainstim conn
Sure that;s what the directory should look like upon access.
Broken:
ls /allen/programs/celltypes/production
0378
And this is what it will look like on a partial expire.
That directory remaining will stop the automount being mounted again and
make it pretty much
useless from this point on.
What we need to do is work out what's causing the partial expire and fix it.
That can be quite difficult.
There usually is some evidence of an error occurring in full debug logs
and I need that.
I can try to duplicate it with your example mount but I suspect it won't
happen when I try it.
I'll have a look at the RHEL revision 102 and see if I can spot anything
too.
Statting of this path results in a wedged terminal, the autofs mount still is showing however:
/etc/auto.programs_celltypes on /allen/programs/celltypes/workgroups/0378 type autofs (rw,relatime,fd=105,pgrp=4816,timeout=0,minproto=5,maxproto=5,offset,pipe_ino=83070
I claim that the autofs mounts should not be expiring at all... and that I want mounted NFS sessions to umount after timeout. Stopping autofs results in a loop of trying to umount the remaining autofs shares until it hits a timeout and gets killed.
The remaining autofs mounts return busy when umount -a -t autofs (or the session locks up). lsof and fuser show 0 processes using the mount point.
4 D root 7929 7805 0 80 0 - 6329 autofs 09:37 pts/2 00:00:00 umount -a -t autofs
I think you'll find that you need to be a bit more careful on umounting
a sub-tree of mounts because of the order umount uses,
If the maps haven't been changed you should be able to umount the tree
in the correct order with "automount --force" provided
automount has been terminated already.
If left alone, the state seems to get continually worse as more autofs mounts expire. Example being on the /allen/programs/celltypes/production path (not a NFS mount point)
ls ing:
/allen/programs/celltypes/production:
^C
Once a partial expire happens it's more or less fatal, all we can do is
try and identify the case
causing it and fix it, if we can.
Logs:
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:21:57 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Jun 5 10:24:27 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Jun 5 10:26:57 n238 automount[25053]: expiring path /allen/programs/celltypes/production
Jun 5 10:26:57 n238 automount[25053]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: expired /allen/programs/celltypes/production
Jun 5 10:26:58 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Don't see any evidence of a problem having occurred here.
Normally I ask for a full debug log (logging = debug in
/etc/autofs.conf) from startup of automount until
the problem occurs. You can send it to me privately if you don't want to
post it.
Ian
