Hi Linux NFS Mailing List, Please excuse any incorrect terminology - I'm not an expert in this space but am learning :). I have recently been experimenting with NFS re-exporting in the 5.11 kernel and have come across an interesting quirk when re-exporting a directory that has subdirectories on different filesystems. I'll do my best to give a concise overview of my setup and the issue I am seeing below. I'll use the following terms to describe the components from here on out: Source NFS Filer: Refers to the originating NFS Server that contains the exports that we want to re-export (Ubuntu 20.04.2 / 5.4.0-1038-gcp). Re-exporting NFS Filer: Refers to the NFS Server that is mounting and re-exporting the Source NFS Filer (Ubuntu 20.10 / 5.11.0-051100-generic). NFS Client: Refers to the server that is mounting the re-exported directories on the Re-exporting NFS Filer (Ubuntu 18.04 / 5.4.0-1036-gcp). (Source NFS Filer <-- Re-exporting NFS Filer <-- NFS Client) --- Source NFS Filer: The source NFS Filer consists of two directories that are each mounted to filesystems on different disks: root@demo-nfs-filer-kernel-community:/home/benmaynard# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 100G 0 disk /files sdc 8:32 0 100G 0 disk /files/disk2 The /files directory is then exported from the source filer, with the crossmnt option set so that clients can move to the child filesystems: root@demo-nfs-filer-kernel-community:/files# cat /etc/exports /files 10.0.0.0/8(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash,crossmnt) -- Re-exporting NFS Filer: >From the Re-exporting NFS Filer, I can then mount this exported directory: root@reexport-server:/home/benmaynard# mount -t nfs -o vers=3,sync 10.67.10.54:/files /files I am able to successfully list the contents of both the /files and /files/disk2 directories: root@reexport-server:/home/benmaynard# ls /files/ a_file disk2 lost+found root@reexport-server:/home/benmaynard# ls /files/disk2/ b_file lost+found Now time to re-export (using the same options as the Source NFS Filer including crossmnt): root@reexport-server:/home/benmaynard# cat /etc/exports /files 10.0.0.0/8(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash,crossmnt) -- NFS Client: I now want to mount the re-exported directory from my NFS Client: root@client-vm:/home/benmaynard# mount -t nfs -o vers=3,sync 10.67.10.51:/files /files The mount completes successfully, but when listing the contents of the directory I receive a Input/output error for the directory that is on a different filesystem: root@client-vm:/home/benmaynard# ls /files ls: cannot access '/files/disk2': Input/output error a_file disk2 lost+found root@client-vm:/home/benmaynard# ls /files/disk2 ls: cannot access '/files/disk2': Input/output error If I unmount and mount the Source NFS Filer directly, I am able to read all of the files. The problem seems to be caused by the re-export. In my efforts to resolve this I tried a number of different options, and the only way I was able to get things working was to explicitly re-export the directories that are on different filesystems from the Re-Exporting NFS Filer (Source NFS Filer exports and NFS Client mount command remains unchanged): root@reexport-server:/home/benmaynard# cat /etc/exports /files 10.0.0.0/8(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash,crossmnt) /files/disk2 10.0.0.0/8(rw,wdelay,no_root_squash,no_subtree_check,fsid=20,sec=sys,rw,secure,no_root_squash,no_all_squash,crossmnt) With these export options I can successfully mount the Re-exporting NFS filer and browse both directores: root@client-vm:/home/benmaynard# mount -t nfs -o vers=3,sync 10.67.10.51:/files /files root@client-vm:/home/benmaynard# ls /files a_file disk2 lost+found root@client-vm:/home/benmaynard# ls /files/disk2/ b_file lost+found It appears as if the issue is something to do with the re-export, but I am struggling to track down the actual cause. This is causing some challenges as the source NFS filer that I am using in production has 1000's of subdirectories on different volumes. Manually re-exporting each of these from our re-export server is a challenge. I have a lab environment configured so happy to do some more testing or run commands if it helps with troubleshooting.
