Anna, Trond, Jeff, Neil, others,
This is a RFC post regarding a problem report from a customer. I am
not sure this is worth fixing though (look towards the end for a few
approaches), and maybe a more experienced NFS developer would have
seen this before and have guidance if this is something worth
pursuing? I think this is likely to have been a long outstanding
issue - the reproducer below works on upstream 6.3-rc5, our RHEL8
series kernels (4.18 based), as well as RHEL7 series (3.10 based)
kernels.
We had a customer report a failure to remove an autofs mountpoint that
was mounted on an underlying NFS mount whose access changed on the NFS
server. Unfortunately the customer was unable to provide the exact
steps of what changed, and was unable to provide a full tcpdump or
kernel trace. However, they did provide the specific 'umount' output
and error, which was unusual (see below), as well as output such as
exports and paths to the underlying mounts. Based on the information
the customer provided, we investigated and were able to replicate the
unique umount error, with a simplified reproducer that does not
involve autofs. The reproducer is attached and output is as follows:
# ./test-non-autofs.sh
setting exports available
exporting 127.0.0.1:/exports/dir1
exporting 127.0.0.1:/exports
setting exports unavailable
exporting 1.2.3.4:/exports/dir1
exporting 1.2.3.4:/exports
sleeping 60s to let attribute cache expire
ls: cannot access '/mnt/exports/dir1': Permission denied
umount.nfs4: /mnt/exports/dir1: block devices not permitted on fs
TEST FAIL on 6.3.0-rc5-bz2149406+
output of 'grep 127.0.0.1 /proc/mounts'
127.0.0.1:/ /mnt/exports nfs4
rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
0 0
127.0.0.1:/dir1 /mnt/exports/dir1 nfs4
rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1
0 0
exporting 127.0.0.1:/exports/dir1
exporting 127.0.0.1:/exports
#
>From ftracing the reproducer (attached), the reason the umount
systemcall fails is due to link_path_walk() failing, which is due to
access being removed on the dependent mountpoint and the NFS attribute
cache expiring. The link_path_walk() will call nfs_permission(), and
ultimately nfs_revalidate_inode() because the attributes have expired.
The nfs_revalidate_inode() will fail due to GETATTR call getting an
NFS server response of AUTH_ERR, which gets sent back up to
user_path_at() with -EACCESS. Since user_path_at() fails, the system
call never gets to path_umount() and thus umount fails.
1915 static int ksys_umount(char __user *name, int flags)
1916 {
1917 int lookup_flags = LOOKUP_MOUNTPOINT;
1918 struct path path;
1919 int ret;
1920
1921 // basic validity checks done first
1922 if (flags & ~(MNT_FORCE | MNT_DETACH | MNT_EXPIRE |
UMOUNT_NOFOLLOW))
1923 return -EINVAL;
1924
1925 if (!(flags & UMOUNT_NOFOLLOW))
1926 lookup_flags |= LOOKUP_FOLLOW;
1927 ret = user_path_at(AT_FDCWD, name, lookup_flags, &path);
<--- this fails with -13 (EACCESS)
1928 if (ret)
1929 return ret;
1930 return path_umount(&path, flags);
1931 }
1932
1933 SYSCALL_DEFINE2(umount, char __user *, name, int, flags)
1934 {
1935 return ksys_umount(name, flags);
1936 }
To fix this, I initially tried to add/reuse an existing mount flag
just as a proof of concept, and fail "user_path_at()" with a special
error code if the flag was set, and continue to path_umount().
However, this approach does not work (kernel oops) because
path_umount() requires a valid path structure, and so user_path_at()
must be successful.
The only other approach I thought about is to somehow pass down the
umount flag all the way down to the NFS layer to nfs_permission(), and
then essentially have nfs_permission() skip over the call to
nfs_revalidate_inode() - the flag would essentially say "there is a
umount in progress, act as though the attributes have not expired on
this directory inode and skip over refreshing". Unfortunately it
looks like I'll need multiple flags at different layers, one that
controls lookup in the VFS layer, and one down to the NFS layer. It's
possible some flags such as LOOKUP_MOUNTPOINT may be re-used, but I'm
not very optimistic about the idea of patching the VFS layer for such
a problem. So in the end, I'm not sure if this is worthwhile, and
would like some feedback on the above before investigating further.
Note that this is the same "cascading mount" configuration that was
reported recently in another recent thread [1] and described in the
patch [2] that fixed a similar problem:
[1] https://lore.kernel.org/linux-nfs/CACH9xG8-tEtWstUVmD9eZFEEAqx-E8Gs14wDL+=uNtBK=-KJvQ@xxxxxxxxxxxxxx/
[2] https://github.com/torvalds/linux/commit/cc89684c9a265828ce061037f1f79f4a68ccd3f7
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
2) | /* nfs_revalidate_inode_enter: fileid=00:2b:59166220 fhandle=0x62d40c52 version=17 */
2) | /* nfs_revalidate_inode_exit: error=-13 (ACCES) fileid=00:2b:59166220 fhandle=0x62d40c52 type=4 (DIR) version=17 size=18 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET) */
------------------------------------------
2) ls-1252 => umount-1253
------------------------------------------
2) | /* nfs_revalidate_inode_enter: fileid=00:2b:59166220 fhandle=0x62d40c52 version=17 */
2) | /* nfs_revalidate_inode_exit: error=-13 (ACCES) fileid=00:2b:59166220 fhandle=0x62d40c52 type=4 (DIR) version=17 size=18 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET) */
3) | /* sys_umount(name: 55aa04c9acc0, flags: 0) */
3) | __x64_sys_umount() {
3) | ksys_umount() {
3) | user_path_at_empty() {
3) | getname_flags() {
3) | getname_flags.part.0() {
3) | kmem_cache_alloc() {
3) 1.077 us | __cond_resched();
3) 0.985 us | should_failslab();
3) 1.010 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.060 us | bpf_ksym_find();
3) 3.046 us | }
3) 1.072 us | filter_irq_stacks();
3) + 19.508 us | }
3) | __check_object_size() {
3) 1.022 us | check_stack_object();
3) 1.013 us | is_vmalloc_addr();
3) 1.022 us | __virt_addr_valid();
3) 1.002 us | __check_heap_object();
3) 8.971 us | }
3) + 31.538 us | }
3) + 33.448 us | }
3) | filename_lookup() {
3) | path_lookupat() {
3) | path_init() {
3) | nd_jump_root() {
3) 1.071 us | set_root();
3) 3.094 us | }
3) 5.119 us | }
3) | link_path_walk.part.0.constprop.0() {
3) | inode_permission() {
3) | generic_permission() {
3) 1.012 us | make_vfsuid();
3) 2.977 us | }
3) | security_inode_permission() {
3) | selinux_inode_permission() {
3) 1.006 us | __inode_security_revalidate();
3) 1.021 us | avc_lookup();
3) 5.042 us | }
3) 6.999 us | }
3) + 12.848 us | }
3) | walk_component() {
3) | lookup_fast() {
3) 1.207 us | __d_lookup_rcu();
3) 3.178 us | }
3) 1.075 us | step_into();
3) 7.143 us | }
3) | inode_permission() {
3) | generic_permission() {
3) 1.036 us | make_vfsuid();
3) 3.002 us | }
3) | security_inode_permission() {
3) | selinux_inode_permission() {
3) 1.087 us | __inode_security_revalidate();
3) 1.313 us | avc_lookup();
3) 5.373 us | }
3) 7.328 us | }
3) + 13.606 us | }
3) | walk_component() {
3) | lookup_fast() {
3) 1.184 us | __d_lookup_rcu();
3) 3.144 us | }
3) | step_into() {
3) 1.057 us | __lookup_mnt();
3) 3.153 us | }
3) 9.160 us | }
3) | inode_permission() {
3) | nfs_permission [nfs]() {
3) | nfs_do_access [nfs]() {
3) | nfs_access_get_cached [nfs]() {
3) 1.175 us | cred_fscmp();
3) 1.591 us | cred_fscmp();
3) 1.110 us | cred_fscmp();
3) 1.375 us | cred_fscmp();
3) 1.258 us | cred_fscmp();
3) 1.112 us | access_cmp [nfs]();
3) | nfs_check_cache_invalid [nfs]() {
3) | nfs_attribute_cache_expired [nfs]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.020 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.123 us | }
3) 5.501 us | }
3) 7.958 us | }
3) 1.043 us | cred_fscmp();
3) 1.043 us | cred_fscmp();
3) 1.056 us | cred_fscmp();
3) 1.084 us | cred_fscmp();
3) 1.068 us | cred_fscmp();
3) 1.050 us | access_cmp [nfs]();
3) | nfs_check_cache_invalid [nfs]() {
3) | nfs_attribute_cache_expired [nfs]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.030 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.069 us | }
3) 5.087 us | }
3) 7.114 us | }
3) + 50.543 us | }
3) + 52.710 us | }
3) + 55.020 us | }
3) + 57.353 us | }
3) | try_to_unlazy() {
3) 1.029 us | legitimize_links();
3) | __legitimize_path() {
3) 1.266 us | __legitimize_mnt();
3) 3.317 us | }
3) | __legitimize_path() {
3) 1.017 us | __legitimize_mnt();
3) 3.038 us | }
3) + 11.397 us | }
3) | inode_permission() {
3) | nfs_permission [nfs]() {
3) | nfs_do_access [nfs]() {
3) | nfs_access_get_cached [nfs]() {
3) 1.063 us | cred_fscmp();
3) 1.051 us | cred_fscmp();
3) 1.042 us | cred_fscmp();
3) 1.043 us | cred_fscmp();
3) 1.077 us | cred_fscmp();
3) 1.053 us | access_cmp [nfs]();
3) | nfs_check_cache_invalid [nfs]() {
3) | nfs_attribute_cache_expired [nfs]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.017 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.054 us | }
3) 5.069 us | }
3) 7.034 us | }
3) 1.046 us | cred_fscmp();
3) 1.048 us | cred_fscmp();
3) 1.047 us | cred_fscmp();
3) 1.048 us | cred_fscmp();
3) 1.066 us | cred_fscmp();
3) 1.040 us | access_cmp [nfs]();
3) | nfs_check_cache_invalid [nfs]() {
3) | nfs_attribute_cache_expired [nfs]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.018 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.037 us | }
3) 5.058 us | }
3) 7.009 us | }
3) | __nfs_revalidate_inode [nfs]() {
3) | /* nfs_revalidate_inode_enter: fileid=00:2b:59166220 fhandle=0x62d40c52 version=17 */
3) 1.022 us | is_bad_inode();
3) | nfs_alloc_fattr_with_label [nfs]() {
3) | kmalloc_trace() {
3) | __kmem_cache_alloc_node() {
3) 1.300 us | __cond_resched();
3) 1.015 us | should_failslab();
3) 5.484 us | }
3) 1.024 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.073 us | bpf_ksym_find();
3) 3.107 us | }
3) 1.195 us | filter_irq_stacks();
3) + 29.668 us | }
3) 1.064 us | nfs4_label_alloc [nfs]();
3) + 33.918 us | }
3) | nfs4_proc_getattr [nfsv4]() {
3) | _nfs4_proc_getattr [nfsv4]() {
3) | nfs4_bitmap_copy_adjust [nfsv4]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.025 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.037 us | }
3) 5.071 us | }
3) 1.077 us | nfs_fattr_init [nfs]();
3) | nfs4_do_call_sync [nfsv4]() {
3) | rpc_run_task [sunrpc]() {
3) + 38.454 us | rpc_new_task [sunrpc]();
3) 3.078 us | rpc_task_set_transport [sunrpc]();
3) # 5152.185 us | rpc_execute [sunrpc]();
3) # 5199.541 us | }
3) | rpc_put_task [sunrpc]() {
3) 1.831 us | rpc_release_resources_task [sunrpc]();
3) + 17.449 us | rpc_free_task [sunrpc]();
3) + 22.347 us | }
3) # 5225.185 us | }
3) # 5235.536 us | }
3) | nfs4_handle_exception [nfsv4]() {
3) 1.301 us | nfs4_do_handle_exception [nfsv4]();
3) 4.323 us | }
3) # 5243.309 us | }
3) | kfree() {
3) | __kmem_cache_free() {
3) 1.026 us | fixup_red_left();
3) 1.044 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.079 us | bpf_ksym_find();
3) 3.083 us | }
3) 1.176 us | filter_irq_stacks();
3) + 21.879 us | }
3) + 23.989 us | }
3) | /* nfs_revalidate_inode_exit: error=-13 (ACCES) fileid=00:2b:59166220 fhandle=0x62d40c52 type=4 (DIR) version=17 size=18 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET) */
3) # 5313.162 us | }
3) # 5356.033 us | }
3) | nfs4_proc_access [nfsv4]() {
3) | _nfs4_proc_access [nfsv4]() {
3) | nfs4_have_delegation [nfsv4]() {
3) 1.033 us | nfs4_is_valid_delegation [nfsv4]();
3) 3.108 us | }
3) | nfs_alloc_fattr [nfs]() {
3) | kmalloc_trace() {
3) | __kmem_cache_alloc_node() {
3) 1.018 us | __cond_resched();
3) 1.014 us | should_failslab();
3) 5.054 us | }
3) 1.025 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.079 us | bpf_ksym_find();
3) 3.080 us | }
3) 1.181 us | filter_irq_stacks();
3) + 27.746 us | }
3) + 29.853 us | }
3) | nfs4_call_sync_sequence [nfsv4]() {
3) | rpc_run_task [sunrpc]() {
3) | rpc_new_task [sunrpc]() {
3) | kmem_cache_alloc() {
3) 1.057 us | __cond_resched();
3) 1.015 us | should_failslab();
3) 1.026 us | get_stack_info();
3) 1.426 us | is_bpf_text_address();
3) 1.183 us | filter_irq_stacks();
3) + 30.917 us | }
3) 1.063 us | xprt_get [sunrpc]();
3) 1.020 us | rpc_task_get_xprt [sunrpc]();
3) 1.063 us | ktime_get();
3) + 39.195 us | }
3) | rpc_task_set_transport [sunrpc]() {
3) | xprt_iter_get_next [sunrpc]() {
3) 1.090 us | xprt_iter_first_entry [sunrpc]();
3) 1.083 us | xprt_get [sunrpc]();
3) 5.219 us | }
3) 1.089 us | rpc_task_get_xprt [sunrpc]();
3) 9.337 us | }
3) | rpc_execute [sunrpc]() {
3) | rpc_make_runnable [sunrpc]() {
3) 1.117 us | wake_up_bit();
3) 3.120 us | }
3) | __rpc_execute [sunrpc]() {
3) 3.104 us | rpc_prepare_task [sunrpc]();
3) 1.029 us | __cond_resched();
3) 1.458 us | call_start [sunrpc]();
3) 1.012 us | __cond_resched();
3) 2.482 us | call_reserve [sunrpc]();
3) 1.013 us | __cond_resched();
3) 1.040 us | call_reserveresult [sunrpc]();
3) 1.014 us | __cond_resched();
3) + 21.795 us | call_refresh [sunrpc]();
3) 1.027 us | __cond_resched();
3) 1.332 us | call_refreshresult [sunrpc]();
3) 1.029 us | __cond_resched();
3) + 20.274 us | call_allocate [sunrpc]();
3) 1.033 us | __cond_resched();
3) + 18.240 us | call_encode [sunrpc]();
3) 1.027 us | __cond_resched();
3) 3.029 us | call_transmit [sunrpc]();
3) ! 467.883 us | out_of_line_wait_on_bit();
3) 1.777 us | call_transmit_status [sunrpc]();
3) 1.043 us | __cond_resched();
3) 1.493 us | call_transmit [sunrpc]();
3) 0.999 us | __cond_resched();
3) 1.606 us | call_transmit_status [sunrpc]();
3) 1.003 us | __cond_resched();
3) 1.061 us | call_bind [sunrpc]();
3) 1.000 us | __cond_resched();
3) + 32.376 us | call_connect [sunrpc]();
3) # 2131.727 us | out_of_line_wait_on_bit();
3) 1.190 us | call_connect_status [sunrpc]();
3) 1.010 us | __cond_resched();
3) 3.242 us | call_transmit [sunrpc]();
3) ! 570.695 us | out_of_line_wait_on_bit();
3) 1.791 us | call_transmit_status [sunrpc]();
3) 1.011 us | __cond_resched();
3) ! 493.953 us | call_transmit [sunrpc]();
3) 1.113 us | __cond_resched();
3) 6.334 us | call_transmit_status [sunrpc]();
3) # 1150.048 us | out_of_line_wait_on_bit();
3) 1.119 us | xprt_timer [sunrpc]();
3) 1.037 us | __cond_resched();
3) 1.150 us | call_status [sunrpc]();
3) 1.032 us | __cond_resched();
3) + 30.391 us | call_decode [sunrpc]();
3) 1.062 us | __cond_resched();
3) 1.539 us | call_encode [sunrpc]();
3) 1.047 us | __cond_resched();
3) 1.443 us | call_transmit [sunrpc]();
3) 1.050 us | __cond_resched();
3) 1.372 us | call_transmit_status [sunrpc]();
3) 1.049 us | __cond_resched();
3) 1.076 us | call_status [sunrpc]();
3) 1.030 us | __cond_resched();
3) 3.770 us | call_decode [sunrpc]();
3) 1.044 us | __cond_resched();
3) 1.403 us | call_encode [sunrpc]();
3) 1.045 us | __cond_resched();
3) 1.342 us | call_transmit [sunrpc]();
3) 1.045 us | __cond_resched();
3) 1.346 us | call_transmit_status [sunrpc]();
3) 1.034 us | __cond_resched();
3) 1.065 us | call_status [sunrpc]();
3) 1.034 us | __cond_resched();
3) 3.768 us | call_decode [sunrpc]();
3) 1.042 us | __cond_resched();
3) 5.643 us | rpc_exit_task [sunrpc]();
3) 1.059 us | __cond_resched();
3) + 57.173 us | rpc_release_resources_task [sunrpc]();
3) # 5154.251 us | }
3) # 5160.417 us | }
3) # 5213.221 us | }
3) | rpc_put_task [sunrpc]() {
3) | rpc_release_resources_task [sunrpc]() {
3) 1.051 us | xprt_release [sunrpc]();
3) | rpc_task_release_client [sunrpc]() {
3) 1.038 us | rpc_task_release_transport [sunrpc]();
3) 3.027 us | }
3) 7.010 us | }
3) | rpc_free_task [sunrpc]() {
3) 1.021 us | put_rpccred [sunrpc]();
3) | mempool_free() {
3) + 14.518 us | mempool_free_slab();
3) + 16.624 us | }
3) + 20.655 us | }
3) + 30.610 us | }
3) # 5247.805 us | }
3) | kfree() {
3) | __kmem_cache_free() {
3) 1.021 us | fixup_red_left();
3) 1.028 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.082 us | bpf_ksym_find();
3) 3.086 us | }
3) 1.166 us | filter_irq_stacks();
3) + 21.878 us | }
3) + 23.945 us | }
3) # 5309.831 us | }
3) | nfs4_handle_exception [nfsv4]() {
3) 1.112 us | nfs4_do_handle_exception [nfsv4]();
3) 3.134 us | }
3) # 5316.254 us | }
3) * 10675.90 us | }
3) * 10677.96 us | }
3) * 10680.02 us | }
3) * 10799.58 us | }
3) | terminate_walk() {
3) | dput() {
3) 1.019 us | __cond_resched();
3) 3.149 us | }
3) | mntput() {
3) 1.238 us | mntput_no_expire();
3) 3.239 us | }
3) | dput() {
3) 1.010 us | __cond_resched();
3) 3.108 us | }
3) | mntput() {
3) 1.089 us | mntput_no_expire();
3) 3.074 us | }
3) + 17.612 us | }
3) * 10826.19 us | }
3) * 10828.24 us | }
3) | putname() {
3) | kmem_cache_free() {
3) 1.014 us | fixup_red_left();
3) 1.020 us | get_stack_info();
3) | is_bpf_text_address() {
3) 1.079 us | bpf_ksym_find();
3) 3.080 us | }
3) 1.096 us | filter_irq_stacks();
3) + 16.295 us | }
3) + 18.389 us | }
3) * 10883.95 us | }
3) | /* ksys_umount: user_path_at failed with -13 */
3) * 10894.07 us | }
3) * 10897.25 us | }
3) | /* sys_umount -> 0xfffffffffffffff3 */
Attachment:
test-non-autofs.sh
Description: application/shellscript
