Summary:
We define a /mount/path/pathcontd/mountpoint tree in autofs, have historically --ghosted the paths, but have recently attempted to set the offset mounts as browse-able.
Our configuration works great... at first. Once the timeout hits (I set this to 10s for debugging... but normally have it set to 300s), the autofs (type) mount points start expiring. Once this expiration starts, the offset mounts disappear (often leaving one mount point remaining that is broken). The disappeared mount points are unable to be re-mounted. This has hit us at semi-regular intervals on Centos 7 , but Rocky 8 servers are highly repro-able. This could be due to the lack of activity on the new systems though.
Config:
distro Name: Rocky Linux 8.8 (Green Obsidian)
autofs-5.1.4-102.el8_8.2
NFS4.{0..2}(though... the nfs mounts are not being invoked when this issue happens.)
Autofs configs:
Auto.master:
/allen/programs/celltypes /etc/auto.programs_celltypes tcp hard intr timeo=600 retrans=2 async --ghost
auto.programs_celltypes(snipping out similar lines to conserve space):
production -fstype=nfs4 /incoming aidc-isi1-prd:/ifs/incoming/aibs/celltypes \
/0378 aidc-isi1-prd:/ifs/programs/celltypes/production/0378 \
/brainstim aidc-isi1-prd:/ifs/programs/celltypes/production/brainstim \
/conn aidc-isi1-prd:/ifs/programs/celltypes/production/conn
workgroups /0285 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0285 \
/0378 aidc-isi1-prd:/ifs/programs/celltypes/workgroups/0378
We have a couple similar mounts in a similar mount point to "/allen/programs/<different name>/<mount>" if it makes a difference there.
What this looks like:
Temporarily working:
ls /allen/programs/celltypes/production
incoming 0378 brainstim conn
Broken:
ls /allen/programs/celltypes/production
0378
Statting of this path results in a wedged terminal, the autofs mount still is showing however:
/etc/auto.programs_celltypes on /allen/programs/celltypes/workgroups/0378 type autofs (rw,relatime,fd=105,pgrp=4816,timeout=0,minproto=5,maxproto=5,offset,pipe_ino=83070
I claim that the autofs mounts should not be expiring at all... and that I want mounted NFS sessions to umount after timeout. Stopping autofs results in a loop of trying to umount the remaining autofs shares until it hits a timeout and gets killed.
The remaining autofs mounts return busy when umount -a -t autofs (or the session locks up). lsof and fuser show 0 processes using the mount point.
4 D root 7929 7805 0 80 0 - 6329 autofs 09:37 pts/2 00:00:00 umount -a -t autofs
If left alone, the state seems to get continually worse as more autofs mounts expire. Example being on the /allen/programs/celltypes/production path (not a NFS mount point)
ls ing:
/allen/programs/celltypes/production:
^C
Logs:
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: expiring path /allen/programs/celltypes/production
Jun 5 10:11:49 n238 automount[19033]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:11:49 n238 automount[19033]: expired /allen/programs/celltypes/production
Jun 5 10:21:57 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Jun 5 10:24:27 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
Jun 5 10:26:57 n238 automount[25053]: expiring path /allen/programs/celltypes/production
Jun 5 10:26:57 n238 automount[25053]: umount_multi: path /allen/programs/celltypes/production incl 1
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_umount_offset: umount offset /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: umounted offset mount /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/brainstim
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/0378
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/humancelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/histformfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/incoming
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfishhisthumanatlas
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousegenetictools
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mousecelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/synapticphys
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/nhpcelltypes
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/wijem
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/whbi
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/mfish
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/general
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/ctyconn
Jun 5 10:26:57 n238 automount[25053]: tree_mapent_delete_offset_tree: deleting offset key /allen/programs/celltypes/production/conn
Jun 5 10:26:57 n238 automount[25053]: expired /allen/programs/celltypes/production
Jun 5 10:26:58 n238 automount[25053]: expire_proc_indirect: expire /allen/programs/celltypes/production
What I have tried:
I was originally distracted by the idea that this was a fileserver issue... testing different versions of NFS etc. But after seeing 0 nfs mounts happening... I am more convinced that this is a bug in my config, or autofs.
The variations of --ghost vs. browse-able have yielded no differences here. The same bug repros on Centos7.9 with an earlier version of autofs.
Setting timeout to 0 (I believe this disables timeout) and this behavior still occurs
