Hi, We’re experiencing some issues when accessing some mounts in a DFS share, which seem to happen since kernel 5.17. After some investigation, we’ve pinpointed the origin to commit a2809d0e16963fdf3984409e47f145 cccb0c6821 - Original BZ for that is https://bugzilla.kernel.org/show_bug.cgi?id=215440 - Patch discussion is at https://patchwork.kernel.org/project/cifs-client/patch/YeHUxJ9zTVNrKveF@xxxxxxxxxxx/ - Similar issues referenced in https://bugzilla.suse.com/show_bug.cgi?id=1198753 Basically, we have a DFS server (attached relevant portion of smb.conf) , Centos 7.9 based, msdfs enabled, consisting of 3 nodes. This server contains a share with a bunch of subfolders, each one of them being a referral to a particular node in the cluster. Since kernel 5.17, when we ask for a file that does not exist, the server seems to give a STATUS_OBJECT_NAME_NOT_FOUND reply, which seems fine, however, this gets processed by the kernel code: [ 4382.120652] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND [ 4382.120675] CIFS: fs/cifs/misc.c: DFS ref '\10.44.7.11\home-DFS\001\notexistingfile.txt' is found, emulate -EREMOTE … [ 4382.128798] CIFS: VFS: cifs_mount failed w/return code = -40 [ 4382.128813] CIFS: fs/cifs/cifs_dfs_ref.c: cifs_dfs_do_automount: cifs_dfs_do_mount:\10.44.7.11\home-DFS\001\notexistingfile.txt , mnt:ffffffffffffffd8 [ 4382.128817] CIFS: fs/cifs/cifs_dfs_ref.c: leaving cifs_dfs_do_automount [ 4382.128818] CIFS: fs/cifs/cifs_dfs_ref.c: leaving cifs_dfs_d_automount [automount failed] … [ 4382.619848] CIFS: fs/cifs/inode.c: Update attributes: \002 inode 0x0000000040d5a675 count 1 dentry: 0x000000000b08edaa d_time 4298866149 jiffies 4299049651 [ 4382.620945] CIFS: Status code returned 0xc0000257 STATUS_PATH_NOT_COVERED [ 4382.620959] CIFS: fs/cifs/inode.c: creating fake fattr for DFS referral Which results in either stat <file> throwing a fake attr structure of a directory item … root@myMachine tmp]# stat pepe/003/notexistingfile.txt File: pepe/003/notexistingfile.txt Size: 0 Blocks: 0 IO Block: 16384 directory Device: 30h/48d Inode: 3322 Links: 2 Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:27:46.598419900 +0100 Modify: 2023-01-16 00:27:46.598419900 +0100 Change: 2023-01-16 00:27:46.598419900 +0100 Birth: - … or the cat command going into a loop , producing -EREMOTE or -40 -ELOOP errors cat: pepe/003/notexistingfile.txt: Too many levels of symbolic links (packet capture / full list of commands and dmesg debug trace attached) Some additional findings... - This seems to happen all the way up to the most recent 6.x kernel, so there hasn't been a fix. - Mounting with -o nodfs still exhibits the same problems for us. - We haven't been able to reproduce any of the issues with e.g. a Windows Server DFS share. - Reverting the above-mentioned commit, by removing the call to the cifs_dfs_query_info_nonascii_quirk function in both connect and inode.c, seems to fix most issues for us. - But even with the code reversion, a query to a file non-existing in the node we're connected to, will still return that the object is remote. The fake-stat / too-many-loops problems disappear, though. So, we have a workaround, but still would like to figure out a complete / permanent solution to this issue. Is there anything else we can try, or should we submit a kernel bug in Bugzilla for this ?
[global] security = ads server max protocol = SMB3 server min protocol = SMB2_02 log level = 0 panic action = winbind use default domain = yes winbind enum users = no winbind enum groups = no load printers = no disable spoolss = yes printing = bsd printcap name = /dev/null host msdfs = yes store dos attributes = yes all shares read only = no num netbios aliases = 32 kernel change notify = no smb2 leases = yes use sendfile = true enable user homes = yes idmap config * : range = 100000-199999999 idmap config * : backend = autorid idmap config * : rangesize = 1000000 realm = xxx netbios name = cluster cluster nodes = node1.xxx node2.xxx node3.xxx cluster ips = 10.44.7.7 10.44.7.9 10.44.7.11 netbios uuid = feb19d01-ac3c-437b-84a3-167a052fd979 max smbd processes = 500 cluster master = node3.xxx server multi channel support = yes workgroup = xxx [home-DFS] vfs objects = shadow_copy streams_depot acl_xattr comment = path = /zroot/shares/%X share uuid = aaa windows_prev_version = yes [othershare] vfs objects = shadow_copy streams_depot acl_xattr comment = path = /zroot/shares/%X/:xxx/yyy msdfs proxy = node3 share uuid = bbb
Attachment:
cifs-fail.pcap
Description: Binary data
Attachment:
workaround.patch
Description: Binary data
[root@myMachine tmp]# mount -t cifs -o backupuid=0,posixpaths,nomapposix,noserverino //10.44.7.7/home-DFS /tmp/pepe [root@myMachine tmp]# ls -l /tmp/pepe total 0 drwx--x--x. 2 root root 0 Jan 16 00:26 001 --> Folders with 711 are local to 10.44.7.7, 755 remote drwx--x--x. 2 root root 0 Jan 16 00:26 002 drwxr-xr-x. 2 root root 0 Jan 16 00:23 003 drwx--x--x. 2 root root 0 Jan 16 00:26 004 drwxr-xr-x. 2 root root 0 May 9 2021 005 drwxr-xr-x. 2 root root 0 May 9 2021 006 drwx--x--x. 2 root root 0 Jan 16 00:26 007 drwx--x--x. 2 root root 0 Jan 16 00:26 008 drwx--x--x. 2 root root 0 Jan 16 00:26 009 drwx--x--x. 2 root root 0 Jan 16 00:26 010 drwxr-xr-x. 2 root root 0 May 9 2021 011 drwx--x--x. 2 root root 0 Jan 16 00:26 012 drwx--x--x. 2 root root 0 Jan 16 00:26 013 drwx--x--x. 2 root root 0 Jan 16 00:26 014 [root@myMachine tmp]# ls pepe/001/existingfile.txt pepe/001/existingfile.txt [root@myMachine tmp]# stat pepe/001/existingfile.txt File: pepe/001/existingfile.txt Size: 0 Blocks: 3 IO Block: 1048576 regular empty file Device: 31h/49d Inode: 3318 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:23:52.248871100 +0100 Modify: 2023-01-16 00:23:38.437777700 +0100 Change: 2023-01-16 00:23:52.488867100 +0100 Birth: 2023-01-16 00:23:52.248871100 +0100 [root@beta-haiku01 tmp]# ls -l pepe/002/existingfile.txt -rwxr-xr-x. 1 root root 0 Jan 16 00:23 pepe/002/existingfile.txt [root@beta-haiku01 tmp]# stat pepe/002/existingfile.txt File: pepe/002/existingfile.txt Size: 0 Blocks: 3 IO Block: 1048576 regular empty file Device: 32h/50d Inode: 3359 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:23:57.461784000 +0100 Modify: 2023-01-16 00:23:38.437777700 +0100 Change: 2023-01-16 00:23:57.803778300 +0100 Birth: 2023-01-16 00:23:57.461784000 +0100 [root@beta-haiku01 tmp]# ls -l pepe/003/existingfile.txt -rwxr-xr-x. 1 root root 0 Jan 16 00:23 pepe/003/existingfile.txt [root@beta-haiku01 tmp]# stat pepe/003/existingfile.txt File: pepe/003/existingfile.txt Size: 0 Blocks: 3 IO Block: 1048576 regular empty file Device: 30h/48d Inode: 3360 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:23:38.437777700 +0100 Modify: 2023-01-16 00:23:38.437777700 +0100 Change: 2023-01-16 00:23:44.833037300 +0100 Birth: 2023-01-16 00:23:38.047968100 +0100 ***All fine, but if we choose to work with non-existing files ... [root@myMachine tmp]# stat pepe/001/notexistingfile.txt File: pepe/001/notexistingfile.txt Size: 0 Blocks: 0 IO Block: 16384 directory Device: 31h/49d Inode: 3319 Links: 2 Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:27:38.185512600 +0100 Modify: 2023-01-16 00:27:38.185512600 +0100 Change: 2023-01-16 00:27:38.185512600 +0100 Birth: - [root@myMachine tmp]# stat pepe/002/notexistingfile.txt File: pepe/002/notexistingfile.txt Size: 0 Blocks: 0 IO Block: 16384 directory Device: 32h/50d Inode: 3321 Links: 2 Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:27:42.403466100 +0100 Modify: 2023-01-16 00:27:42.403466100 +0100 Change: 2023-01-16 00:27:42.403466100 +0100 Birth: - [root@myMachine tmp]# stat pepe/003/notexistingfile.txt File: pepe/003/notexistingfile.txt Size: 0 Blocks: 0 IO Block: 16384 directory Device: 30h/48d Inode: 3322 Links: 2 Access: (0711/drwx--x--x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:cifs_t:s0 Access: 2023-01-16 00:27:46.598419900 +0100 Modify: 2023-01-16 00:27:46.598419900 +0100 Change: 2023-01-16 00:27:46.598419900 +0100 Birth: - [root@myMachine tmp]# cat pepe/001/notexistingfile.txt cat: pepe/001/notexistingfile.txt: Too many levels of symbolic links [root@myMachine tmp]# cat pepe/002/notexistingfile.txt cat: pepe/002/notexistingfile.txt: Too many levels of symbolic links [root@myMachine tmp]# cat pepe/003/notexistingfile.txt cat: pepe/003/notexistingfile.txt: Too many levels of symbolic links
Attachment:
cifsFYItrace.txt.gz
Description: application/gzip
