On Thu, Oct 07, 2021 at 10:38:45AM -0700, dai.ngo@xxxxxxxxxx wrote: > > On 10/7/21 10:03 AM, dai.ngo@xxxxxxxxxx wrote: > > > >On 9/23/21 3:39 PM, dai.ngo@xxxxxxxxxx wrote: > >> > >>On 9/23/21 2:50 PM, Bruce Fields wrote: > >>>On Thu, Jul 15, 2021 at 04:45:22PM -0700, dai.ngo@xxxxxxxxxx wrote: > >>>>Hi Bruce, > >>>Oops, sorry for neglecting this. > >>> > >>>>I'm doing some locking testing between NFSv4 and SMB client and > >>>>think there are some issues on the server that allows both clients > >>>>to lock the same file at the same time. > >>>It's not too surprising to me that getting consistent locks between the > >>>two would be hard. > >>> > >>>Did you get any review from a Samba expert? I seem to recall it having > >>>a lot of options, and I wonder if it's configured correctly for this > >>>case. > >> > >>No, I have not heard from any Samba expert. > >> > >>> > >>>It sounds like Samba may be giving out oplocks without getting a lease > >>>from the kernel. > >> > >>I will have to circle back to this when we're done with the 1st > >>phase of courteous server. > > > >I disabled oplock for the SMB share and locking between NFSv4 and SMB > >client works as expected. It appears that smbd does not set the VFS > >lease on the file after granting oplock to smb client. > > Enabling kernel oplocks has the same effect, smbd does not grant oplock > to client forcing it to send lock request. OK, good, so that's working as expected. I understand that there are still some deficiencies in the kernel lease implementation, but I'm not sure how to hit those cases with this kind of testing. --b. > > -Dai > > > > >-Dai > > > >>-Dai > >> > >>> > >>>--b. > >>> > >>>>Here is what I did: > >>>> > >>>>NOTE: lck is a simple program that use lockf(3) to lock a file from > >>>>offset 0 to the length specified by '-l'. > >>>> > >>>>On NFSv4 client > >>>>--------------- > >>>> > >>>>[root@nfsvmd07 ~]# nfsstat -m > >>>>/tmp/mnt from nfsvmf24:/root/smb_share > >>>>Flags: > >>>>rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard, > >>>>proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.80.62.47, > >>>> local_lock=none,addr=10.80.111.94 > >>>>[root@nfsvmd07 ~]# > >>>> > >>>> > >>>>[root@nfsvmd07 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000 > >>>>Lck/file: 1, Maxlocks: 10000000 > >>>>Locking[/tmp/mnt/messages] Offset[0] Len[100000000] > >>>>N[0]...doing F_LOCK.. > >>>>LOCKED... > >>>> > >>>>Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop > >>>> > >>>>[NFS client successfully locks the file] > >>>> > >>>>On SMB client > >>>>------------- > >>>> > >>>>[root@nfsvme24 ~]# mount |grep cifs > >>>>//nfsvmf24/smb_share on /tmp/mnt type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=0,noforcegid,addr=10.80.111.94,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1) > >>>>[root@nfsvme24 ~]# > >>>> > >>>>[root@nfsvme24 ~]# smbclient -L nfsvmf24 > >>>>Enter SAMBA\root's password: > >>>> > >>>> Sharename Type Comment > >>>> --------- ---- ------- > >>>> print$ Disk Printer Drivers > >>>> smb_share Disk Test Samba Share <<===== share to mount > >>>> IPC$ IPC IPC Service (Samba 4.10.16) > >>>> root Disk Home Directories > >>>>Reconnecting with SMB1 for workgroup listing. > >>>> > >>>> Server Comment > >>>> --------- ------- > >>>> > >>>> Workgroup Master > >>>> --------- ------- > >>>>[root@nfsvme24 ~]# > >>>> > >>>>[root@nfsvme24 ~]# ./lck -p /tmp/mnt/messages -W -l 100000000 > >>>>Lck/file: 1, Maxlocks: 10000000 > >>>>Locking[/tmp/mnt/messages] Offset[0] Len[100000000] > >>>>N[0]...doing F_LOCK.. > >>>>LOCKED... > >>>> > >>>>Locks[1] files[1] took[2.000s] sleep waiting...Hit Control-C to stop > >>>> > >>>>[SMB client successfully locks the file] > >>>> > >>>>The same issue happens when either client locks the file first. > >>>>I think this is what has happened: > >>>> > >>>>1. NFSv4 client opens and locks the file first > >>>> > >>>> . NFSv4 client send OPEN and LOCK to server, server replies > >>>> OK on both requests. > >>>> > >>>> . SMB client sends create request with Oplock==Lease for > >>>> the same file. > >>>> > >>>> . server holds off on replying to SMB client's create request, > >>>> recalls delegation from NFSv4 client, waits for NFSv4 client > >>>> to return the delegation then replies success to SMB client's > >>>> create request with lease granted (Oplock==Lease). > >>>> > >>>> NOTE: I think SMB server should replies the create request > >>>> with Oplock==None to force the SMB client to sends the > >>>> lock request. > >>>> > >>>> . Once SMB client receives the reply of the create with > >>>> 'Oplock==Lease', it assumes it has full control of the file > >>>> therefor it does not need to send the lock request. > >>>> > >>>> . both NFSv4 and SMB client now think they have locked the file. > >>>> > >>>>pcap: nfs_lock_smb_lock.pcap > >>>> > >>>>2. SMB client creates the file with 'Oplock==Lease' first > >>>> > >>>> . SMB sends create request with 'Oplock==Lease' to server, > >>>> server replies OK with 'Oplock==Lease'. SMB client skips > >>>> sending lock request since it assumes it has full control > >>>> of the file with the lease. > >>>> > >>>> . NFSv4 client sends OPEN to server, server replies OK with > >>>> delagation is none. NFSv4 client sends LOCK request, since > >>>> no lock was created in the kernel for the SMB client, the > >>>> lock was granted to the NFSv4 client. > >>>> > >>>> NOTE: I think the SMB server should send lease break > >>>> notification to the SMB client, wait for the lease break > >>>> acknowledgment from SMB client before replying to the > >>>> OPEN of the NFSv4 client. This will force the SMB client > >>>> to send the lock request to the server. > >>>> > >>>> . both NFSv4 and SMB client now think they have locked the file. > >>>> > >>>>Your thought? > >>>> > >>>>Thanks, > >>>> > >>>>-Dai
