On Thu, Sep 22, 2022 at 1:48 PM Jan Kara <jack@xxxxxxx> wrote: > > On Tue 20-09-22 21:19:25, Amir Goldstein wrote: [...] > > Hi Jan, > > > > I wanted to give an update on the POC that I am working on. > > I decided to find a FUSE HSM and show how it may be converted > > to use fanotify HSM hooks. > > > > HTTPDirFS is a read-only FUSE filesystem that lazyly populates a local > > cache from a remote http on first access to every directory and file range. > > > > Normally, it would be run like this: > > ./httpdirfs --cache-location /vdf/cache https://cdn.kernel.org/pub/ /mnt/pub/ > > > > Content is accessed via FUSE mount as /mnt/pub/ and FUSE implements > > passthrough calls to the local cache dir if cache is already populated. > > > > After my conversion patches [1], this download-only HSM can be run like > > this without mounting FUSE: > > > > sudo ./httpdirfs --fanotify --cache-location /vdf/cache > > https://cdn.kernel.org/pub/ - > > > > [1] https://github.com/amir73il/httpdirfs/commits/fanotify_pre_content > > > > Browsing the cache directory at /vdf/cache, lazyly populates the local cache > > using FAN_ACCESS_PERM readdir hooks and lazyly downloads files content > > using FAN_ACCESS_PERM read hooks. > > > > Up to this point, the implementation did not require any kernel changes. > > However, this type of command does not populate the path components, > > because lookup does not generate FAN_ACCESS_PERM event: > > > > stat /vdf/cache/data/linux/kernel/firmware/linux-firmware-20220815.tar.gz > > > > To bridge that functionality gap, I've implemented the FAN_LOOKUP_PERM > > event [2] and used it to lazyly populate directories in the path ancestry. > > For now, I stuck with the XXX_PERM convention and did not require > > FAN_CLASS_PRE_CONTENT, although we probably should. > > > > [2] https://github.com/amir73il/linux/commits/fanotify_pre_content > > > > Streaming read of large files works as well, but only for sequential read > > patterns. Unlike the FUSE read calls, the FAN_ACCESS_PERM events > > do not (yet) carry range info, so my naive implementation downloads > > one extra data chunk on each FAN_ACCESS_PERM until the cache file is full. > > > > This makes it possible to run commands like: > > > > tar tvfz /vdf/cache/data/linux/kernel/firmware/linux-firmware-20220815.tar.gz > > | less > > > > without having to wait for the entire 400MB file to download before > > seeing the first page. > > > > This streaming feature is extremely important for modern HSMs > > that are often used to archive large media files in the cloud. > > Thanks for update Amir! I've glanced through the series and so far it looks > pretty simple and I'd have only some style / readability nits (but let's > resolve those once we have something more complete). > > When thinking about HSM (and while following your discussion with Dave) I > wondered about one thing: When the notifications happen before we take > locks, then we are in principle prone to time-to-check-time-to-use races, > aren't we? How are these resolved? > > For example something like: > We have file with size 16k. > Reader: Writer > read 8k at offset 12k > -> notification sent > - HSM makes sure 12-16k is here and 16-20k is beyond eof so nothing to do > > expand file to 20k > - now the file contents must not get moved out until the reader is > done in order not to break it > Hi Jan, It's been a while since I updated this topic. I have been making progress on the wiki and POC, but it's not done yet. I would like to poke your brain about my proposed solutions for the TOCTOU race issues, because the solution is subtle and you may have better ideas to suggest. > > > Questions: > > - What do you think about the direction this POC has taken so far? > > - Is there anything specific that you would like to see in the POC > > to be convinced that this API will be useful? > > I think your POC is taking a good direction and your discussion with Dave > had made me more confident that this is all workable :). I liked your idea > of the wiki (or whatever form of documentation) that summarizes what we've > discussed in this thread. That would be actually pretty nice for future > reference. > The current state of POC is that "populate of access" of both files and directories is working and "race free evict of file content" is also implemented (safely AFAIK). The technique involving exclusive write lease is discussed at [1]. In a nutshell, populate and evict synchronize on atomic i_writecount and this technique can be implemented with upstream UAPIs. I did use persistent xattr marks for the POC, but this is not a must. Evictable inode marks would have worked just as well. Now I have started to work on persistent change tracking. For this, I have only kernel code, only lightly tested, but I did not prove yet that the technique is working. The idea that I started to sketch at [2] is to alternate between two groups. When a change is recorded, an evictable ignore mark will be added on the object. To start recording changes from a new point in time (checkpoint), a new group will be created (with no ignore marks) and the old group will be closed. The core of the algorithm is the "safe handover" between groups. This requires two infrastructure additions. The first is FAN_MARK_SYNC [3] as described in commit message: --- Synchronous add of mark or remove/flush of marks with ignore mask provides a method for safe handover of event handling between two groups: - First, group A subscribes to some events with FAN_MARK_SYNC - Then, group B unsubscribes from those events This method guarantees that any event that both groups subscribed to, will be delivered to either group or to both of them. Note that FAN_MARK_SYNC provides no synchronization to the object interest masks, which are checked outside srcu read side. Therefore, this method does not provide any guarantee regarding delivery of events which only one of the groups is subscribed to. For example, if only group B was subscribed to FAN_OPEN_EXEC and only group A is subscribing only to FAN_OPEN, an execution of a binary file may not deliver FAN_OPEN_EXEC to group B nor FAN_OPEN to group A. --- The second is to overlap fsnotify_mark_srcu read side with sb_start_write(), for pre modify permission events [4] as described in commit message: --- fsnotify: acquire sb write access inside pre modify permission event For pre modify permission events, acquire sb write access before leaving SRCU and return >0 to signal that sb write access was acquired. This can be used to implement safe "handover" of pre modify permission events between two fanotify groups: - First, group A subscribes to pre modify events with FAN_MARK_SYNC - Then, a freeze/thaw cycle is performed on the filesystem - Finally, group B unsubscribes from those events This method guarantees that a pre modify event that both groups subscribed to will be delivered to either group or to both of them. In case that the pre modify event is delivered only to group B, the freeze/thaw cycle guarantees that the filesystem modification that followed that pre modify event was also completed, before the handover is complete and group B can be closed. For pre rename permission event, acquire sb write access after the second of the event pair (i.e. rename to) was authorized. --- What do you think about this handover technique? Do you think that it is workable or do you see any major flaws in it? Would you use a different or an additional synchronization primitive instead of abusing fsnotify_mark_srcu? To clarify, the race that I am trying to avoid is: 1. group B got a pre modify event and recorded the change before time T 2. The actual modification is performed after time T 3. group A does not get a pre modify event, so does not record the change in the checkpoint since T Thanks, Amir. [1] https://github.com/amir73il/fsnotify-utils/wiki/Hierarchical-Storage-Management-API#invalidating-local-cache [2] https://github.com/amir73il/fsnotify-utils/wiki/Hierarchical-Storage-Management-API#tracking-local-modifications [3] https://github.com/amir73il/linux/commits/fan_mark_sync [4] https://github.com/amir73il/linux/commits/fan_modify_perm
