On Fri, Dec 15, 2023 at 5:31 PM Josef Bacik <josef@xxxxxxxxxxxxxx> wrote:
>
> On Wed, Dec 13, 2023 at 09:09:30PM +0200, Amir Goldstein wrote:
> > On Wed, Dec 13, 2023 at 7:28 PM Jan Kara <jack@xxxxxxx> wrote:
> > >
> > > On Fri 08-12-23 10:01:35, Amir Goldstein wrote:
> > > > With FAN_DENY response, user trying to perform the filesystem operation
> > > > gets an error with errno set to EPERM.
> > > >
> > > > It is useful for hierarchical storage management (HSM) service to be able
> > > > to deny access for reasons more diverse than EPERM, for example EAGAIN,
> > > > if HSM could retry the operation later.
> > > >
> > > > Allow userspace to response to permission events with the response value
> > > > FAN_DENY_ERRNO(errno), instead of FAN_DENY to return a custom error.
> > > >
> > > > The change in fanotify_response is backward compatible, because errno is
> > > > written in the high 8 bits of the 32bit response field and old kernels
> > > > reject respose value with high bits set.
> > > >
> > > > Signed-off-by: Amir Goldstein <amir73il@xxxxxxxxx>
> > >
> > > So a couple of comments that spring to my mind when I'm looking into this
> > > now (partly maybe due to my weak memory ;):
> > >
> > > 1) Do we still need the EAGAIN return? I think we have mostly dealt with
> > > freezing deadlocks in another way, didn't we?
> >
> > I was thinking about EAGAIN on account of the HSM not being able to
> > download the file ATM.
> >
> > There are a bunch of error codes that are typical for network filesystems, e.g.
> > ETIMEDOUT, ENOTCONN, ECONNRESET which could be relevant to
> > HSM failures.
> >
> > >
> > > 2) If answer to 1) is yes, then there is a second question - do we expect
> > > the errors to propagate back to the unsuspecting application doing say
> > > read(2) syscall? Because I don't think that will fly well with a big
> > > majority of applications which basically treat *any* error from read(2) as
> > > fatal. This is also related to your question about standard permission
> > > events. Consumers of these error numbers are going to be random
> > > applications and I see a potential for rather big confusion arising there
> > > (like read(1) returning EINVAL or EBADF and now you wonder why the hell
> > > until you go debug the kernel and find out the error is coming out of
> > > fanotify handler). And the usecase is not quite clear to me for ordinary
> > > fanotify permission events (while I have no doubts about creativity of
> > > implementors of fanotify handlers ;)).
> > >
> >
> > That's a good question.
> > I prefer to delegate your question to the prospect users of the feature.
> >
> > Josef, which errors did your use case need this feature for?
> >
> > > 3) Given the potential for confusion, maybe we should stay conservative and
> > > only allow additional EAGAIN error instead of arbitrary errno if we need it?
> > >
> >
> > I know I was planning to use this for EDQUOT error (from FAN_PRE_MODIFY),
> > but I certainly wouldn't mind restricting the set of custom errors.
> > I think it makes sense. The hard part is to agree on this set of errors.
> >
>
> I'm all for flexibility here.
>
> We're going to have 2 classes of applications interacting with HSM backed
> storage, normal applications and applications that know they're backed by HSM.
> The normal applications are just going to crash if they get an error on read(2),
> it doesn't matter what errno it is. The second class would have different
> things they'd want to do in the face of different errors, and that's what this
> patchset is targeting. We can limit it to a few errno's if that makes you feel
> better, but having more than just one would be helpful.
>
Ok. In another email I got from your colleagues, they listed:
EIO, EAGAIN, ENOSPC as possible errors to return.
I added EDQUOT for our in house use case.
Those are all valid errors for write(2) and some are valid for read(2).
ENOSPC/EDQUOT make a lot of sense for HSM for read(2), but could
be surprising to applications not aware of HSM.
I think it wouldn't be that bad if we let HSM decide which of those errors
to return for FAN_PRE_ACCESS as opposed to FAN_PRE_MODIFY.
But given that we do want to limit the chance of abusing this feature,
perhaps it would be best to limit the error codes to known error codes
for write(2) IO failures (i.e. not EBADF, not EFAULT) and allow returning
FAN_DENY_ERRNO only for the new FAN_PRE_{ACCESS,MODIFY}
HSM events.
IOW, FAN_{OPEN,ACCESS}_PERM - no FAN_DENY_ERRNO for you!
Does that sound good to you?
Furthermore, we can start with allowing a very limited set of errors
and extend it in the future, on case by case basis.
The way that this could be manageable is if we provide userspace
a way to test for supported return codes.
There is already a simple method that we used for testing FAN_INFO
records type support -
After fan_fd = fanotify_init(), userspace can write a "test" fanotify_response
to fan_fd with fanotify_response.fd=FAN_NOFD.
When setting fanotify_response.fd=FAN_DENY, this would return ENOENT,
but with fanotify_response.fd=FAN_DENY_ERRNO(EIO), upstream would
return EINVAL.
This opens the possibility of allowing, say, EIO, EAGAIN in the first release
and ENOSPC, EDQUOT in the following release.
The advantage in this method is that it is very simple and already working
correctly for old kernels.
The downside is that this simple method does not allow checking for allowed
errors per specific event type, so if we decide that we do want to
allow returning
FAN_DENY_ERRNO for FAN_OPEN_PERM later on, this method could not
be used by userspace to test for this finer grained support.
In another thread, I mention the fact that FAN_OPEN_PERM still has a
potential freeze deadlock when called from open(O_TRUNC|O_CREATE),
so we can consider the fact that FAN_DENY_ERRNO is not allowed with
FAN_OPEN_PERM as a negative incentive for people to consider using
FAN_OPEN_PERM as a trigger for HSM.
Thoughts?
Thanks,
Amir.
