> Hang on, fuse does use caches in the kernel (page cache, > dcache/icache). The issue is probably not lack of cache, it's how the > caches are primed and used. Did you disable these caches? Did you > not disable invalidation for data, metadata and dcache? In recent > kernels we added caching readdir as well. The only objects not cached > are (non-acl) xattrs. Do you have those? Android (which is our primary use case) is constantly under memory pressure, so caches don't actually last long. Our experience with FOPEN_KEEP_CACHE has shown that pages are evicted more often than the files are getting reopened, so it doesn't help. FUSE has to re-read the data from the backing store all the time. We didn't use xattrs for the FUSE-based implementation, but ended up requiring a similar thing in the Incremental FS, so the final design would have to include them. > Re prefetching data: > there's the NOTIFY_STORE message. To add to the previous point, we do not have the data for prefetching, as we're loading it page-by-page from the host. We had to disable readahead for FUSE completely, otherwise even USB3 isn't fast enough to deliver data in that big chunks in time, and applications keep hanging on page faults. Overall, better caching doesn't save much *on Android*; what would work is a full-blown data storage system inside FUSE kernel code, that can intercept requests before they go into user mode and process them completely. That's how we could keep the data out of RAM but still get rid of that extra context switch and kernel-user transition. But this also means that FUSE becomes damn too much aware of the specific storage format and all its features, and basically gets specialized implementation of one of its filesystem inside the generic FUSE code. Even if we separate that out, the kernel API between the storage and FUSE ended up being complete VFS API copy, with some additions to send data blocks and Merkle tree blocks in. The code is truly if we stuff the Incremental FS into FUSE instead of mounting it directly. -- Thanks, Yurii
