On Wed, Nov 1, 2017 at 8:49 AM, Chuck Lever <chuck.lever@xxxxxxxxxx> wrote: > >> On Nov 1, 2017, at 10:53 AM, Pradeep <pradeepthomas@xxxxxxxxx> wrote: >> >> Adding linux-nfs (did not work last couple of times because of email format). >> >> Is this supposed to work with Linux NFS clients (see the problem >> description at the end of this email)? > > Yes. I've used referrals successfully with upstream kernels in the > past week (against non-Linux servers, even). > > Looks like this is a RHEL issue, though. Should you report the > issue to Red Hat? > You can easily reproduce this on Ubuntu 16.04 as well (we have tried up to 4.9.37). > >> The NFSv4 referrals with Linux clients does not work with 'stat', 'ls' >> etc., But linux client follows referrals after a 'cd'. Is this the >> expected behavior? > > I think so. "ls -l" in the parent directory isn't going to trigger > a mount, but "cd" will. After the mount, the mounted on directory > on the client will appear as expected with "ls -l". > 'ls -l' will show incorrect stats if referral is not followed. Client doesn't seem to use attributes from READDIR (or it throws away that). stat shows output like this for referral directories - you can see from tcpdump that server(nfs-ganesha) sent the attributes correctly. $ stat /mnt/dir.0 File: ‘/mnt/dir.0’ Size: 0 Blocks: 0 IO Block: 1048576 directory Device: 26h/38d Inode: 1085 Links: 2 Access: (0555/dr-xr-xr-x) Uid: (4294967294/ UNKNOWN) Gid: (4294967294/ UNKNOWN) Context: system_u:object_r:nfs_t:s0 Access: 1969-12-31 16:00:00.000000000 -0800 Modify: 1969-12-31 16:00:00.000000000 -0800 Change: 1969-12-31 16:00:00.000000000 -0800 Birth: - > >> tcpdump is attached. > > Traffic to the destination server might be going over a different > network interface. Check your tcpdump command line. > I have only one network interface. > You could also enable NFS and/or RPC debugging (before reproducing) > to see the steps taken by the client displayed in /var/log/messages. > > # rpcdebug -m nfs -s > # rpcdebug -m rpc -s > debug messages from /var/log/messages is attached (see readdir.log and stat.log). The 'ls -l' output is below. 'dir.0' is the referral directory. $ ls -l /mnt total 0 dr-xr-xr-x. 2 4294967294 4294967294 0 Dec 31 1969 dir.0 drwxrwxr-x. 2 pradeep pradeep 6 Oct 16 17:07 dir.1 drwxrwxr-x. 2 pradeep pradeep 6 Oct 16 17:07 dir.2 The problem appears to be in the code path below: nfs4_proc_lookup_common -> _nfs4_proc_lookup -> nfs4_get_referral -> nfs_fixup_referral_attributes /* Fixup attributes for the nfs_lookup() call to nfs_fhget() */ nfs_fixup_referral_attributes(&locations->fattr); /* replace the lookup nfs_fattr with the locations nfs_fattr */ memcpy(fattr, &locations->fattr, sizeof(struct nfs_fattr)); 'fattr' will never have attributes other than FSID and fs_locations. > >>>> On Mon, Oct 30, 2017 at 3:24 PM, Frank Filz <ffilzlnx@xxxxxxxxxxxxxx> >>>> wrote: >>>>> >>>>> Oh, I had forgotten about that patch… >>>>> >>>>> >>>>> >>>>> Can you try any other clients? This may be a client issue (I did see some >>>>> suspicious code in the client). >>>>> >>>>> >>>>> >>>>> It may also be that you need a fully qualified path (starting with a /). >>>>> >>>>> >>>>> >>>>> It looks like Ganesha is doing the right thing though. >>>>> >>>>> >>>>> >>>>> Frank >>>>> >>>>> >>>>> >>>>> From: Pradeep [mailto:pradeep.thomas@xxxxxxxxx] >>>>> Sent: Monday, October 30, 2017 2:21 PM >>>>> To: Frank Filz <ffilzlnx@xxxxxxxxxxxxxx> >>>>> Cc: nfs-ganesha-devel <nfs-ganesha-devel@xxxxxxxxxxxxxxxxxxxxx>; >>>>> ssaurabh.wisc@xxxxxxxxx >>>>> Subject: Re: [Nfs-ganesha-devel] NFSv4 referrals not working with >>>>> ganesha. >>>>> >>>>> >>>>> >>>>> Hi Frank, >>>>> >>>>> >>>>> >>>>> This is with latest version of Ganesha. The referral support is already >>>>> in VFS: https://review.gerrithub.io/c/353684 >>>>> >>>>> >>>>> >>>>> tcpdump is attached. From the tcpdump, we can see that the stat sent a >>>>> LOOKUP for the remote export and received a moved error. It also sent back >>>>> the fs_locations. But the client (CentOS 7.3) never followed that with a >>>>> LOOKUP to the remote server. >>>>> >>>>> >>>>> >>>>> You can see that packet #41 has the correct FS locations. But client does >>>>> not do another lookup to get the correct attributes. >>>>> >>>>> >>>>> >>>>> $ stat /mnt/nfs_d1 >>>>> >>>>> File: ‘/mnt/nfs_d1’ >>>>> >>>>> Size: 0 Blocks: 0 IO Block: 1048576 directory >>>>> >>>>> Device: 28h/40d Inode: 1 Links: 2 >>>>> >>>>> Access: (0555/dr-xr-xr-x) Uid: (4294967294/ UNKNOWN) Gid: (4294967294/ >>>>> UNKNOWN) >>>>> >>>>> Context: system_u:object_r:nfs_t:s0 >>>>> >>>>> Access: 1969-12-31 16:00:00.000000000 -0800 >>>>> >>>>> Modify: 1969-12-31 16:00:00.000000000 -0800 >>>>> >>>>> Change: 1969-12-31 16:00:00.000000000 -0800 >>>>> >>>>> Birth: - >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Oct 30, 2017 at 12:13 PM, Frank Filz <ffilzlnx@xxxxxxxxxxxxxx> >>>>> wrote: >>>>> >>>>> What version of Ganesha? I assume by “native” FSAL, you mean FSAL_VFS? >>>>> Did you add the fs locations XATTR support? FSAL_GPFS currently has the only >>>>> in-tree referral support and I’m not sure it necessarily works, but I’m >>>>> unable to test it. >>>>> >>>>> >>>>> >>>>> If you have code for FSAL_VFS to add the fs locations attribute, go ahead >>>>> and post it and I could poke at it. >>>>> >>>>> >>>>> >>>>> Also, tcpdump traces might help understand what is going wrong. >>>>> >>>>> >>>>> >>>>> Frank >>>>> >>>>> >>>>> >>>>> From: Pradeep [mailto:pradeep.thomas@xxxxxxxxx] >>>>> Sent: Monday, October 30, 2017 11:45 AM >>>>> To: nfs-ganesha-devel <nfs-ganesha-devel@xxxxxxxxxxxxxxxxxxxxx> >>>>> Cc: ssaurabh.wisc@xxxxxxxxx >>>>> Subject: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha. >>>>> >>>>> >>>>> >>>>> Hi all, >>>>> >>>>> >>>>> >>>>> We are testing NFSv4 referral for Linux CentOS 7 with nfs-ganesha and are >>>>> running >>>>> >>>>> into some serious issues. >>>>> >>>>> >>>>> >>>>> Although, we were able to set up NFSv4 referral using the native Ganesha >>>>> FSAL, >>>>> >>>>> we could not get it fully functional for all Linux client system calls. >>>>> >>>>> Basically, the NFSv4 spec suggests to return a NFS4ERR_MOVED on a >>>>> >>>>> LOOKUP done for a remote export. However, this breaks the `stat` system >>>>> call on >>>>> >>>>> Linux CentOS 7 (stat’ results in a LOOKUP,GETFH,GETATTR compound). An >>>>> easy way to >>>>> >>>>> reproduce the broken behavior is: >>>>> >>>>> 1) mount the root of the pseudo file system and >>>>> >>>>> 2) issue a `stat` command on the remote export. >>>>> >>>>> The stat returned are corrupt. >>>>> >>>>> >>>>> >>>>> After digging into the CentOS 7 client code, we realized that the stat >>>>> operation >>>>> >>>>> is never expected to follow the referral. However, switching to returning >>>>> NFS4_OK >>>>> >>>>> for stat, then breaks `cd` or a `ls -l` command, because now we don't >>>>> know when >>>>> >>>>> to follow the referral. >>>>> >>>>> >>>>> >>>>> Does anyone have a successful experience in setting up the NFSv4 >>>>> referrals that >>>>> >>>>> they could share? Or, if some suggestions on what we might be doing >>>>> wrong? >>>>> >>>>> >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> Virus-free. www.avast.com >>>>> >>>>> >>>> >>>> >>> >> <nfs_remote_export1.pcap> > > -- > Chuck Lever > > >
Attachment:
readdir.log
Description: Binary data
Attachment:
stat.log
Description: Binary data