Thanks for all of the resources! I was trying to implement an NFS server, and v3 sounded like an easier place to start :-) I think I'll move on to v4. If we're revisiting the past, maybe just one last historical question: Do either of you know why the Linux Kernel only uses the IP address/svid to identify the caller? FreeBSD uses the owner field as well. Jan On Sun, Aug 7, 2022 at 8:01 AM Tom Talpey <tom@xxxxxxxxxx> wrote: > > On 8/6/2022 3:49 PM, Trond Myklebust wrote: > > On Sat, 2022-08-06 at 11:03 -0400, Jan Kasiak wrote: > >> Hi Trond, > >> > >> The v4 RFCs do mention protocol design flaws, but don't go into more > >> detail. > >> > >> I was trying to understand those flaws in order to understand how and > >> why v3 was problematic. > >> > >> > > > > The main issues derive from the fact that NLM is a side band protocol, > > meaning that it has no ability to influence the NFS protocol > > operations. In particular, there is no way to ensure safe ordering of > > locks and I/O. e.g. if your readahead code kicks in while you are > > unlocking the file, then there is nothing that guarantees the page > > reads happened while the lock was in place on the server. > > The same weakness also causes problems for reboots: if your client > > doesn't notice that the server rebooted (and lost your locks) because > > the statd callback mechanism failed, then you're SOL. Your I/O may > > succeed, but can end up causing problems for another client that has > > since grabbed the lock and assumes it now has exclusive access to the > > file. > > > > NLM also suffers from intrinsic problems of its own such as lack of > > only-once semantics. If you send a blocking LOCK request, and > > subsequently send a CANCEL operation, then who knows whether or not the > > lock or the cancel get processed first by the server? Many servers will > > reply LCK_GRANTED to the CANCEL even if they did not find the lock > > request. Sending an UNLOCK can also cause issues if the lock was > > granted via a blocking lock callback (NLM_GRANTED) since there is no > > ordering between the reply to the NLM_GRANTED and the UNLOCK. > > > > Finally, as already mentioned, there are multiple issues associated > > with client or server reboot. The NLM mechanism is pretty dependent on > > yet another side band mechanism (STATD) to tell you when this occurs, > > but that mechanism does not work to release the locks held by a client > > if it fails to come back after reboot. Even if the client does come > > back, it might forget to invoke the statd process, or it might use a > > different identifier than it did during the last boot instance (e.g. > > because DHCP allocated a different IP address, or the IP address it not > > unique due to use of NAT, or a hostname was used that is non-unique, > > ...). > > If the server reboots, then it may fail to notify the client of that > > reboot through the callback mechanism. Reasons may include the > > existence of a NAT, failure of the rpcbind/portmapper process on the > > client, firewalls,... > > That brought back memories. > > http://www.nfsv4bat.org/Documents/ConnectAThon/2006/talpey-cthon06-nsm.pdf > > Here's an even older issues list for nlm on Solaris circa 1996. > The portrait-mode slides are in reverse order. :) > > http://www.nfsv4bat.org/Documents/ConnectAThon/1996/lockmgr.pdf > > The NLM protocol is an antique and hasn't been looked at in well > over a decade (or two!). NLMv4 (circa 1995) widened offsets to > 64-bit, which was the last innovation it got. None of the RPC > sideband protocols were ever standardized, btw. > > Jan, what are you planning to use it for? Personally I'd advise > against pretty much anything. > > Tom. > > > > >> -Jan > >> > >> > >> On Fri, Aug 5, 2022 at 10:27 PM Trond Myklebust > >> <trondmy@xxxxxxxxxxxxxxx> wrote: > >>> > >>> On Fri, 2022-08-05 at 19:17 -0400, Jan Kasiak wrote: > >>>> Hi, > >>>> > >>>> I was looking at the code for nlmclnt_lock and wanted to ask a > >>>> question about how the Linux kernel client and the NLM 4 protocol > >>>> handle some errors around certain edge cases. > >>>> > >>>> Specifically, I think there is a race condition around two > >>>> threads of > >>>> the same program acquiring a lock, one of the threads being > >>>> interrupted, and the NFS client sending an unlock when none of > >>>> the > >>>> program threads called unlock. > >>>> > >>>> On NFS server machine S: > >>>> there exists an unlocked file F > >>>> > >>>> On NFS client machine C: > >>>> in program P: > >>>> thread 1 tries to lock(F) with fd A > >>>> thread 2 tries to lock(F) with fd B > >>>> > >>>> The Linux client will issue two NLM_LOCK calls with the same svid > >>>> and > >>>> same range, because it uses the program id to map to an svid. > >>>> > >>>> For whatever reason, assume the connection is broken (cable gets > >>>> pulled etc...) > >>>> and `status = nlmclnt_call(cred, req, NLMPROC_LOCK);` fails. > >>>> > >>>> The Linux client will retry the request, but at some point thread > >>>> 1 > >>>> receives a signal and nlmclnt_lock breaks out of its loop. > >>>> Because > >>>> the > >>>> Linux client request failed, it will fall through and go to the > >>>> out_unlock label, where it will want to send an unlock request. > >>>> > >>>> Assume that at some point the connection is reestablished. > >>>> > >>>> The Linux kernel client now has two outstanding lock requests to > >>>> send > >>>> to the remote server: one for a lock that thread 2 is still > >>>> trying to > >>>> acquire, and one for an unlock of thread 1 that failed and was > >>>> interrupted. > >>>> > >>>> I'm worried that the Linux client may first send the lock > >>>> request, > >>>> and > >>>> tell thread 2 that it acquired the lock, and then send an unlock > >>>> request from the cancelled thread 1 request. > >>>> > >>>> The server will successfully process both requests, because the > >>>> svid > >>>> is the same for both, and the true server side state will be that > >>>> the > >>>> file is unlocked. > >>>> > >>>> One can talk about the wisdom of using multiple threads to > >>>> acquire > >>>> the > >>>> same file lock, but this behavior is weird, because none of the > >>>> threads called unlock. > >>>> > >>>> I have experimented with reproducing this, but have not been > >>>> successful in triggering this ordering of events. > >>>> > >>>> I've also looked at the code of in clntproc.c and I don't see a > >>>> spot > >>>> where outstanding failed lock/unlock requests are checked while > >>>> processing lock requests? > >>>> > >>>> Thanks, > >>>> -Jan > >>> > >>> Nobody here is likely to want to waste much time trying to 'fix' > >>> the > >>> NLM locking protocol. The protocol itself is known to be extremely > >>> fragile, and the endemic problems constitute some of the main > >>> motivations for the development of the NFSv4 protocol > >>> (See https://datatracker.ietf.org/doc/html/rfc2624#section-8 > >>> and https://datatracker.ietf.org/doc/html/rfc7530#section-9). > >>> > >>> If you need more reliable support for POSIX locks beyond what > >>> exists > >>> today for NLM, then please consider NFSv4. > >>> > >>> -- > >>> Trond Myklebust > >>> Linux NFS client maintainer, Hammerspace > >>> trond.myklebust@xxxxxxxxxxxxxxx > >>> > >>> > >
