On Mon 15-04-24 17:23:40, Amir Goldstein wrote: > On Mon, Mar 4, 2024 at 12:33 PM Jan Kara <jack@xxxxxxx> wrote: > > On Tue 27-02-24 21:42:37, Amir Goldstein wrote: > > > On Mon, Feb 19, 2024 at 1:01 PM Jan Kara <jack@xxxxxxx> wrote: > > > > > > These handlers would return EACCESS unless > > > > > > there's somebody handling events and returning something else. > > > > > > > > > > > > You could then do: > > > > > > > > > > > > fan_fd = fanotify_init() > > > > > > ffd = fsopen() > > > > > > fsconfig(ffd, FSCONFIG_SET_STRING, "source", device, 0) > > > > > > fsconfig(ffd, FSCONFIG_SET_FLAG, "hsm", NULL, 0) > > > > > > rootfd = fsconfig(ffd, FSCONFIG_CMD_CREATE, NULL, NULL, 0) > > > > > > fanotify_mark(fan_fd, FAN_MARK_ADD, ... , rootfd, NULL) > > > > > > <now you can move the superblock into the mount hierarchy> > > > > > > > > > > Not too bad. > > > > > I think that "hsm_deny_mask=" mount options would give more flexibility, > > > > > but I could be convinced otherwise. > > > > > > > > > > It's probably not a great idea to be running two different HSMs on the same > > > > > fs anyway, but if user has an old HSM version installed that handles only > > > > > pre-content events, I don't think that we want this old version if it happens > > > > > to be run by mistake, to allow for unsupervised create,rename,delete if the > > > > > admin wanted to atomically mount a fs that SHOULD be supervised by a > > > > > v2 HSM that knows how to handle pre-path events. > > > > > > > > > > IOW, and "HSM bit" on sb is too broad IMO. > > > > > > > > OK. So "hsm_deny_mask=" would esentially express events that we require HSM > > > > to handle, the rest would just be accepted by default. That makes sense. > > > > > > Yes. > > > > > > > The only thing I kind of dislike is that this ties fanotify API with mount > > > > API. So perhaps hsm_deny_mask should be specified as a string? Like > > > > preaccess,premodify,prelookup,... and transformed into a bitmask only > > > > inside the kernel? It gives us more maneuvering space for the future. > > > > > > > > > > Urgh. I see what you are saying, but this seems so ugly to me. > > > I have a strong feeling that we are trying to reinvent something > > > and that we are going to reinvent it badly. > > > I need to look for precedents, maybe in other OS. > > > I believe that in Windows, there is an API to register as a > > > Cloud Engine Provider, so there is probably a way to have multiple HSMs > > > working on different sections of the filesystem in some reliable > > > crash safe manner. > > > > OK, let's see what other's have came up with :) > > From my very basic Google research (did not ask Chat GPT yet ;)) > I think that MacOS FSEvents do not have blocking permission events at all, > so there is no built-in HSM API. > > The Windows Cloud Sync Engine API: > https://learn.microsoft.com/en-us/windows/win32/cfapi/build-a-cloud-file-sync-engine > Does allow registring different "Storage namespace providers". > AFAICT, the persistence of "Place holder" files is based on NTFS > "Reparse points", > which are a long time native concept which allows registering a persistent > hook on a file to be handled by a specific Windows driver. > > So for example, a Dropbox place holder file, is a file with "reparse point" > that has some label to direct the read/write calls to the Windows > Cloud Sync Engine > driver and a sub-label to direct the handling of the upcall by the Dropbox > CloudSync Engine service. OK, so AFAIU they implement HSM directly in the filesystem which is somewhat different situation from what we are trying to do. > I do not want to deal with "persistent fanotify marks" at this time, so > maybe something like: > > fsconfig(ffd, FSCONFIG_SET_STRING, "hsmid", "dropbox", 0) > fsconfig(ffd, FSCONFIG_SET_STRING, "hsmver", "1", 0) > > Add support ioctls in fanotify_ioctl(): > - FANOTIFY_IOC_HSMID > - FANOTIFY_IOC_HSMVER What would these do? Set HSMID & HSMVER for fsnotify_group identified by 'file'? BTW I'm not so convinced about the 'version' thing. I have hard time to remember an example where the versioning in the API actually ended up being useful. I also expect tight coupling between userspace mounting the filesystem and setting up HSM so it is hard to imagine some wrong version of HSM provider would be "accidentally" started for the filesystem. > And require that a group with matching hsmid and recent hsmver has a live > filesystem mark on the sb. I'm not quite following here. We'd require matching fsnotify group for what? For mounting the fs? For not returning EPERM from all pre-op handlers? Either way that doesn't make sense to me as its unclear how userspace would be able to place the mark... But there's a way around that - since the HSM app will have its private non-HSM mount for filling in contents, it can first create that mount, place filesystem marks though it and then remount the superblock with hsmid mount option and create the public mount. But I'm not sure if you meant this or something else... > If this is an acceptable API for a single crash-safe HSM provider, then the > question becomes: > How would we extend this to multiple crash-safe HSM providers in the future? Something like: fsconfig(ffd, FSCONFIG_SET_STRING, "hsmid", "dropbox,cloudsync,httpfs", 0) means all of them are required to have a filesystem mark? > Or maybe we do not need to support multiple HSM groups per sb? > Maybe in the future a generic service could be implemented to > delegate different HSM modules, e.g.: > > fsconfig(ffd, FSCONFIG_SET_STRING, "hsmid", "cloudsync", 0) > fsconfig(ffd, FSCONFIG_SET_STRING, "hsmver", "1", 0) > > And a generic "cloudsync" service could be in charge of > registration of "cloudsync" engines and dispatching the pre-content > event to the appropriate module based on path (i.e. under the dropbox folder). > > Once this gets passed NACKs from fs developers I'd like to pull in > some distro people to the discussion and maybe bring this up as > a topic discussion for LSFMM if we feel that there is something to discuss. I guess a short talk (lighting talk?) what we are planning to do would be interesting so that people are aware. At this point I don't think we have particular problems to discuss that would be interesting for the whole fs crowd for a full slot... Honza -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR
