> On Jan 26, 2024, at 4:53 AM, gaurav gangalwar <gaurav.gangalwar@xxxxxxxxx> wrote: > > Hi, > I had one query with NFS rdma support, details mentioned in the mail, but couldn't reach out to nfs-rdma-devel@xxxxxxxxxxxxxxxxxxxxx > Please check if this is the right mailing list to ask this question, else point me to the correct mailing list. > > Thanks, > Gaurav Gangalwar > > ---------- Forwarded message --------- > From: gaurav gangalwar <gaurav.gangalwar@xxxxxxxxx> > Date: Fri, Jan 26, 2024 at 3:14 PM > Subject: NFS4.0 rdma with referal > To: <nfs-rdma-devel@xxxxxxxxxxxxxxxxxxxxx> > > > Hello, > I was experimenting with Linux nfs kernel rdma with nfsref. > With direct mount NFSv4.0 it works fine with rdma > But when I add a referral using nfsref, it uses tcp for referral mount. > > [root@uvm-ca102ba ~]# sudo mount -vvv -o rdma,port=20049,vers=4.0 10.53.65.28:/expdir rdma-kernel > mount.nfs: timeout set for Fri Jan 26 01:28:40 2024 > mount.nfs: trying text-based options 'rdma,port=20049,vers=4.0,addr=10.53.65.28,clientaddr=10.51.43.103' > [root@uvm-ca102ba ~]# cd rdma-kernel > [root@uvm-ca102ba rdma-kernel]# find . > . > ./expdir1 > [root@uvm-ca102ba rdma-kernel]# nfsstat -m > /root/rdma-kernel from 10.53.65.28:/expdir > Flags: rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.28 > > /root/rdma-kernel/expdir1 from 10.53.65.36:/expdir1 > Flags: rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.36 > > > I am using SoftROCE and both the ips have a rdma port added for nfs. > > ~$ cat /proc/fs/nfsd/portlist > rdma 20049 > rdma 20049 > udp 2049 > tcp 2049 > udp 2049 > tcp 2049 > > Attaching tcpdump also for same > > Why is the client not using rdma to mount referral fs location? This is only speculation. The client would need to negotiate the transport setting if it wants to try rdma. It can't assume that a referral that it encounters on an proto=rdma mount will redirect to a server that also has RDMA. I don't think the Linux NFS client does that negotiating yet. -- Chuck Lever
