On Thu, 2024-01-25 at 14:02 +0000, David Howells wrote: > Here's a roadmap for the future development of netfslib and local caching > (e.g. cachefiles). > > Netfslib > ======== > > [>] Current state: > > The netfslib write helpers have gone upstream now and are in v6.8-rc1, with > both the 9p and afs filesystems using them. This provides larger I/O size > support to 9p and write-streaming and DIO support to afs. > > The helpers provide their own version of generic_perform_write() that: > > (1) doesn't use ->write_begin() and ->write_end() at all, completely taking > over all of of the buffered I/O operations, including writeback. > > (2) can perform write-through caching, setting up one or more write > operations and adding folios to them as we copy data into the pagecache > and then starting them as we finish. This is then used for O_SYNC and > O_DSYNC and can be used with immediate-write caching modes in, say, cifs. > > Filesystems using this then deal with iov_iters and ideally would not deal > pages or folios at all - except incidentally where a wrapper is necessary. > > > [>] Aims for the next merge window: > > Convert cifs to use netfslib. This is now in Steve French's for-next branch. > > Implement content crypto and bounce buffering. I have patches to do this, but > it would only be used by ceph (see below). > > Make libceph and rbd use iov_iters rather than referring to pages and folios > as much as possible. This is mostly done and rbd works - but there's one bit > in rbd that still needs doing. > > Convert ceph to use netfslib. This is about half done, but there are some > wibbly bits in the ceph RPCs that I'm not sure I fully grasp. I'm not sure > I'll quite manage this and it might get bumped. > > Finally, change netfslib so that it uses ->writepages() to write data to the > cache, even data on clean pages just read from the server. I have a patch to > do this, but I need to move cifs and ceph over first. This means that > netfslib, 9p, afs, cifs and ceph will no longer use PG_private_2 (aka > PG_fscache) and Willy can have it back - he just then has to wrest control > from NFS and btrfs. > > > [>] Aims for future merge windows: > > Using a larger chunk size than PAGE_SIZE - for instance 256KiB - but that > might require fiddling with the VM readahead code to avoid read/read races. > > Cache AFS directories - there are just files and currently are downloaded and > parsed locally for readdir and lookup. > > Cache directories from other filesystems. > > Cache inode metadata, xattrs. > > Add support for fallocate(). > > Implement content crypto in other filesystems, such as cifs which has its own > non-fscrypt way of doing this. > > Support for data transport compression. > > Disconnected operation. > > NFS. NFS at the very least needs to be altered to give up the use of > PG_private_2. > > > Local Caching > ============= > > There are a number of things I want to look at with local caching: > > [>] Although cachefiles has switched from using bmap to using SEEK_HOLE and > SEEK_DATA, this isn't sufficient as we cannot rely on the backing filesystem > optimising things and introducing both false positives and false negatives. > Cachefiles needs to track the presence/absence of data for itself. > > I had a partially-implemented solution that stores a block bitmap in an xattr, > but that only worked up to files of 1G in size (with bits representing 256K > blocks in a 512-byte bitmap). > > [>] An alternative cache format might prove more fruitful. Various AFS > implementations use a 'tagged cache' format with an index file and a bunch of > small files each of which contains a single block (typically 256K in OpenAFS). > > This would offer some advantages over the current approach: > > - it can handle entry reuse within the index > - doesn't require an external culling process > - doesn't need to truncate/reallocate when invalidating > > There are some downsides, including: > > - each block is in a separate file > - metadata coherency is more tricky - a powercut may require a cache wipe > - the index key is highly variable in size if used for multiple filesystems > > But OpenAFS has been using this for something like 30 years, so it's probably > worth a try. > > [>] Need to work out some way to store xattrs, directory entries and inode > metadata efficiently. > > [>] Using NVRAM as the cache rather than spinning rust. > > [>] Support for disconnected operation to pin desirable data and keep > track of changes. > > [>] A user API by which the cache for specific files or volumes can be > flushed. > > > Disconnected Operation > ====================== > > I'm working towards providing support for disconnected operation, so that, > provided you've got your working set pinned in the cache, you can continue to > work on your network-provided files when the network goes away and resync the > changes later. > > This is going to require a number of things: > > (1) A user API by which files can be preloaded into the cache and pinned. > > (2) The ability to track changes in the cache. > > (3) A way to synchronise changes on reconnection. > > (4) A way to communicate to the user when there's a conflict with a third > party change on reconnect. This might involve communicating via systemd > to the desktop environment to ask the user to indicate how they'd like > conflicts recolved. > > (5) A way to prompt the user to re-enter their authentication/crypto keys. > > (6) A way to ask the user how to handle a process that wants to access data > we don't have (error/wait) - and how to handle the DE getting stuck in > this fashion. > > David > This is all great stuff, David! Would it be reasonable to request a slot to talk about the state of all of this at LSF/MM in May? -- Jeff Layton <jlayton@xxxxxxxxxx>
