Hi Patrick, On Wed, Aug 4, 2021 at 2:17 PM Patrick Goetz <pgoetz@xxxxxxxxxxxxxxx> wrote: > > > > On 8/3/21 9:10 PM, Trond Myklebust wrote: > > > > > > On Tue, 2021-08-03 at 21:51 -0400, Matt Benjamin wrote: > >> (who have performed an open) > >> > >> On Tue, Aug 3, 2021 at 9:43 PM Matt Benjamin <mbenjami@xxxxxxxxxx> > >> wrote: > >>> > >>> I think it is how close-to-open has been traditionally understood. > >>> I > >>> do not believe that close-to-open in any way implies a single > >>> writer, > >>> rather it sets the consistency expectation for all readers. > >>> > > > > OK. I'll bite, despite the obvious troll-bait... > > > > > > close-to-open implies a single writer because it is impossible to > > guarantee ordering semantics in RPC. You could, in theory, do so by > > serialising on the client, but none of us do that because we care about > > performance. > > > > If you don't serialise between clients, then it is trivial (and I'm > > seriously tired of people who whine about this) to reproduce reads to > > file areas that have not been fully synced to the server, despite > > having data on the client that is writing. i.e. the reader sees holes > > that never existed on the client that wrote the data. > > The reason is that the writes got re-ordered en route to the server, > > and so reads to the areas that have not yet been filled are showing up > > as holes. > > > > So, no, the close-to-open semantics definitely apply to both readers > > and writers. > > > > So, I have a naive question. When a client is writing to cache, why > wouldn't it be possible to send an alert to the server indicating that > the file is being changed. The server would keep track of such files > (client cached, updated) and act accordingly; i.e. sending a request to > the client to flush the cache for that file if another client is asking > to open the file? The process could be bookended by the client alerting > the server when the cached version has been fully synchronized with the > copy on the server so that the server wouldn't serve that file until the > synchronization is complete. The only problem I can see with this is the > client crashing or disconnecting before the file is fully written to the > server, but then some timeout condition could be set. We already have this! What you're describing is almost exactly how delegations work :) Anna > > > > >>> Matt > >>> > >>> On Tue, Aug 3, 2021 at 5:36 PM bfields@xxxxxxxxxxxx > >>> <bfields@xxxxxxxxxxxx> wrote: > >>>> > >>>> On Tue, Aug 03, 2021 at 09:07:11PM +0000, Trond Myklebust wrote: > >>>>> On Tue, 2021-08-03 at 16:30 -0400, J. Bruce Fields wrote: > >>>>>> On Fri, Jul 30, 2021 at 02:48:41PM +0000, Trond Myklebust > >>>>>> wrote: > >>>>>>> On Fri, 2021-07-30 at 09:25 -0400, Benjamin Coddington > >>>>>>> wrote: > >>>>>>>> I have some folks unhappy about behavior changes after: > >>>>>>>> 479219218fbe > >>>>>>>> NFS: > >>>>>>>> Optimise away the close-to-open GETATTR when we have > >>>>>>>> NFSv4 OPEN > >>>>>>>> > >>>>>>>> Before this change, a client holding a RO open would > >>>>>>>> invalidate > >>>>>>>> the > >>>>>>>> pagecache when doing a second RW open. > >>>>>>>> > >>>>>>>> Now the client doesn't invalidate the pagecache, though > >>>>>>>> technically > >>>>>>>> it could > >>>>>>>> because we see a changeattr update on the RW OPEN > >>>>>>>> response. > >>>>>>>> > >>>>>>>> I feel this is a grey area in CTO if we're already > >>>>>>>> holding an > >>>>>>>> open. > >>>>>>>> Do we > >>>>>>>> know how the client ought to behave in this case? Should > >>>>>>>> the > >>>>>>>> client's open > >>>>>>>> upgrade to RW invalidate the pagecache? > >>>>>>>> > >>>>>>> > >>>>>>> It's not a "grey area in close-to-open" at all. It is very > >>>>>>> cut and > >>>>>>> dried. > >>>>>>> > >>>>>>> If you need to invalidate your page cache while the file is > >>>>>>> open, > >>>>>>> then > >>>>>>> by definition you are in a situation where there is a write > >>>>>>> by > >>>>>>> another > >>>>>>> client going on while you are reading. You're clearly not > >>>>>>> doing > >>>>>>> close- > >>>>>>> to-open. > >>>>>> > >>>>>> Documentation is really unclear about this case. Every > >>>>>> definition of > >>>>>> close-to-open that I've seen says that it requires a cache > >>>>>> consistency > >>>>>> check on every application open. I've never seen one that > >>>>>> says "on > >>>>>> every open that doesn't overlap with an already-existing open > >>>>>> on that > >>>>>> client". > >>>>>> > >>>>>> They *usually* also preface that by saying that this is > >>>>>> motivated by > >>>>>> the > >>>>>> use case where opens don't overlap. But it's never made > >>>>>> clear that > >>>>>> that's part of the definition. > >>>>>> > >>>>> > >>>>> I'm not following your logic. > >>>> > >>>> It's just a question of what every source I can find says close- > >>>> to-open > >>>> means. E.g., NFS Illustrated, p. 248, "Close-to-open consistency > >>>> provides a guarantee of cache consistency at the level of file > >>>> opens and > >>>> closes. When a file is closed by an application, the client > >>>> flushes any > >>>> cached changs to the server. When a file is opened, the client > >>>> ignores > >>>> any cache time remaining (if the file data are cached) and makes > >>>> an > >>>> explicit GETATTR call to the server to check the file > >>>> modification > >>>> time." > >>>> > >>>>> The close-to-open model assumes that the file is only being > >>>>> modified by > >>>>> one client at a time and it assumes that file contents may be > >>>>> cached > >>>>> while an application is holding it open. > >>>>> The point checks exist in order to detect if the file is being > >>>>> changed > >>>>> when the file is not open. > >>>>> > >>>>> Linux does not have a per-application cache. It has a page > >>>>> cache that > >>>>> is shared among all applications. It is impossible for two > >>>>> applications > >>>>> to open the same file using buffered I/O, and yet see different > >>>>> contents. > >>>> > >>>> Right, so based on the descriptions like the one above, I would > >>>> have > >>>> expected both applications to see new data at that point. > >>>> > >>>> Maybe that's not practical to implement. It'd be nice at least > >>>> if that > >>>> was explicit in the documentation. > >>>> > >>>> --b. > >>>> > >>> > >>> > >>> -- > >>> > >>> Matt Benjamin > >>> Red Hat, Inc. > >>> 315 West Huron Street, Suite 140A > >>> Ann Arbor, Michigan 48103 > >>> > >>> http://www.redhat.com/en/technologies/storage > >>> > >>> tel. 734-821-5101 > >>> fax. 734-769-8938 > >>> cel. 734-216-5309 > >> > >> > >> > >
