John Spray <jspray@xxxxxxxxxx> 2015-10-19 11:34:
<snip/>
CephFS supports capabilities to manages access to objects, enforce
consistency of data etc. IMHO a sane way to handle the page cache is use a
capability to inform the mds about caches objects; as long as no other
client claims write access to an object or its metadata, the cache copy is
considered consistent. Upon write access the client should drop the
capability (and thus remove the object from the page cache). If another
process tries to access a cache object with intact 'cache' capability, it
may be promoted to read/write capability.
This is essentially what we already do, except that we pro-actively
drop the capability when files are closed, rather than keeping it
around on the client in case its needed again.
Having those caps linger on a client is a tradeoff:
* while it makes subsequent cached reads from the original client
nice and fast, it adds latency for any other client that wants to open
the file.
* It also adds latency for the original client when it wants to open
many other files, because it will have to wait for the original file's
capabilities to be given up before it has room in its metadata cache
to open other files.
* it also creates confusion if someone opens a big file, then closes
it, then wonders why their ceph-fuse process is still sitting on gigs
of memory
This doesn't really address any of the cache coherency issues raised on
the thread, but I wonder if hooking into something like the cachefs
facility would alleviate the memory concerns raise above?
The idea would basically be to just store the cached cephfs blocks on a
local fs and let the regular cachefs/localfs facilities handle keeping
those pages in the pagecache or not as necessary. In the case that a
file is reopened on a client later on, it still would need to
reestablish its cache validity with the server(s) but then could
(hopefully) just get the data from the local cachefs which (hopefully)
may also still have it still in the local pagecache. If not, all you've
done is a few local OS calls which shouldn't be too bad compared to the
network IO required to fetch the full file contents again from the
server(s). That seems too simple so there must be another wrinkle in
there somewhere, right?
The only thread I was able to find on something like that in the past
appears to have died in kernel panics [1].
Thanks,
Brian
[1] <https://www.redhat.com/archives/linux-cachefs/2013-May/msg00049.html>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
