On Tue, Aug 27, 2024 at 1:12 AM Jingbo Xu <jefflexu@xxxxxxxxxxxxxxxxx> wrote:
>
> On 8/24/24 6:54 AM, Joanne Koong wrote:
> > On Thu, Aug 22, 2024 at 7:17 PM Jingbo Xu <jefflexu@xxxxxxxxxxxxxxxxx> wrote:
> >>
> >> On 8/23/24 5:19 AM, Joanne Koong wrote:
> >>> On Thu, Aug 22, 2024 at 12:06 AM Jingbo Xu <jefflexu@xxxxxxxxxxxxxxxxx> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 8/14/24 7:22 AM, Joanne Koong wrote:
> >>>>> Introduce two new sysctls, "default_request_timeout" and
> >>>>> "max_request_timeout". These control timeouts on replies by the
> >>>>> server to kernel-issued fuse requests.
> >>>>>
> >>>>> "default_request_timeout" sets a timeout if no timeout is specified by
> >>>>> the fuse server on mount. 0 (default) indicates no timeout should be enforced.
> >>>>>
> >>>>> "max_request_timeout" sets a maximum timeout for fuse requests. If the
> >>>>> fuse server attempts to set a timeout greater than max_request_timeout,
> >>>>> the system will default to max_request_timeout. Similarly, if the max
> >>>>> default timeout is greater than the max request timeout, the system will
> >>>>> default to the max request timeout. 0 (default) indicates no timeout should
> >>>>> be enforced.
> >>>>>
> >>>>> $ sysctl -a | grep fuse
> >>>>> fs.fuse.default_request_timeout = 0
> >>>>> fs.fuse.max_request_timeout = 0
> >>>>>
> >>>>> $ echo 0x100000000 | sudo tee /proc/sys/fs/fuse/default_request_timeout
> >>>>> tee: /proc/sys/fs/fuse/default_request_timeout: Invalid argument
> >>>>>
> >>>>> $ echo 0xFFFFFFFF | sudo tee /proc/sys/fs/fuse/default_request_timeout
> >>>>> 0xFFFFFFFF
> >>>>>
> >>>>> $ sysctl -a | grep fuse
> >>>>> fs.fuse.default_request_timeout = 4294967295
> >>>>> fs.fuse.max_request_timeout = 0
> >>>>>
> >>>>> Signed-off-by: Joanne Koong <joannelkoong@xxxxxxxxx>
> >>>>> Reviewed-by: Josef Bacik <josef@xxxxxxxxxxxxxx>
> >>>>> Reviewed-by: Bernd Schubert <bschubert@xxxxxxx>
> >>>>> ---
> >>>>> Documentation/admin-guide/sysctl/fs.rst | 17 ++++++++++
> >>>>> fs/fuse/Makefile | 2 +-
> >>>>> fs/fuse/fuse_i.h | 16 ++++++++++
> >>>>> fs/fuse/inode.c | 19 ++++++++++-
> >>>>> fs/fuse/sysctl.c | 42 +++++++++++++++++++++++++
> >>>>> 5 files changed, 94 insertions(+), 2 deletions(-)
> >>>>> create mode 100644 fs/fuse/sysctl.c
> >>>>>
> >>>>> diff --git a/Documentation/admin-guide/sysctl/fs.rst b/Documentation/admin-guide/sysctl/fs.rst
> >>>>> index 47499a1742bd..44fd495f69b4 100644
> >>>>> --- a/Documentation/admin-guide/sysctl/fs.rst
> >>>>> +++ b/Documentation/admin-guide/sysctl/fs.rst
> >>>>> @@ -332,3 +332,20 @@ Each "watch" costs roughly 90 bytes on a 32-bit kernel, and roughly 160 bytes
> >>>>> on a 64-bit one.
> >>>>> The current default value for ``max_user_watches`` is 4% of the
> >>>>> available low memory, divided by the "watch" cost in bytes.
> >>>>> +
> >>>>> +5. /proc/sys/fs/fuse - Configuration options for FUSE filesystems
> >>>>> +=====================================================================
> >>>>> +
> >>>>> +This directory contains the following configuration options for FUSE
> >>>>> +filesystems:
> >>>>> +
> >>>>> +``/proc/sys/fs/fuse/default_request_timeout`` is a read/write file for
> >>>>> +setting/getting the default timeout (in seconds) for a fuse server to
> >>>>> +reply to a kernel-issued request in the event where the server did not
> >>>>> +specify a timeout at mount. 0 indicates no timeout.
> >>>>> +
> >>>>> +``/proc/sys/fs/fuse/max_request_timeout`` is a read/write file for
> >>>>> +setting/getting the maximum timeout (in seconds) for a fuse server to
> >>>>> +reply to a kernel-issued request. If the server attempts to set a
> >>>>> +timeout greater than max_request_timeout, the system will use
> >>>>> +max_request_timeout as the timeout. 0 indicates no timeout.
> >>>>
> >>>> "0 indicates no timeout"
> >>>>
> >>>> I think 0 max_request_timeout shall indicate that there's no explicit
> >>>> maximum limitation for request_timeout.
> >>>
> >>> Hi Jingbo,
> >>>
> >>> Ah I see where the confusion in the wording is (eg that "0 indicates
> >>> no timeout" could be interpreted to mean there is no timeout at all
> >>> for the connection, rather than no timeout as the max limit). Thanks
> >>> for pointing this out. I'll make this more explicit in v5. I'll change
> >>> the wording above for the "default_request_timeout" case too.
> >>>
> >>>>
> >>>>
> >>>>> diff --git a/fs/fuse/Makefile b/fs/fuse/Makefile
> >>>>> index 6e0228c6d0cb..cd4ef3e08ebf 100644
> >>>>> --- a/fs/fuse/Makefile
> >>>>> +++ b/fs/fuse/Makefile
> >>>>> @@ -7,7 +7,7 @@ obj-$(CONFIG_FUSE_FS) += fuse.o
> >>>>> obj-$(CONFIG_CUSE) += cuse.o
> >>>>> obj-$(CONFIG_VIRTIO_FS) += virtiofs.o
> >>>>>
> >>>>> -fuse-y := dev.o dir.o file.o inode.o control.o xattr.o acl.o readdir.o ioctl.o
> >>>>> +fuse-y := dev.o dir.o file.o inode.o control.o xattr.o acl.o readdir.o ioctl.o sysctl.o
> >>>>> fuse-y += iomode.o
> >>>>> fuse-$(CONFIG_FUSE_DAX) += dax.o
> >>>>> fuse-$(CONFIG_FUSE_PASSTHROUGH) += passthrough.o
> >>>>> diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
> >>>>> index 0a2fa487a3bf..dae9977fa050 100644
> >>>>> --- a/fs/fuse/fuse_i.h
> >>>>> +++ b/fs/fuse/fuse_i.h
> >>>>> @@ -47,6 +47,14 @@
> >>>>> /** Number of dentries for each connection in the control filesystem */
> >>>>> #define FUSE_CTL_NUM_DENTRIES 5
> >>>>>
> >>>>> +/*
> >>>>> + * Default timeout (in seconds) for the server to reply to a request
> >>>>> + * if no timeout was specified on mount
> >>>>> + */
> >>>>> +extern u32 fuse_default_req_timeout;
> >>>>> +/** Max timeout (in seconds) for the server to reply to a request */
> >>>>> +extern u32 fuse_max_req_timeout;
> >>>>> +
> >>>>> /** List of active connections */
> >>>>> extern struct list_head fuse_conn_list;
> >>>>>
> >>>>> @@ -1486,4 +1494,12 @@ ssize_t fuse_passthrough_splice_write(struct pipe_inode_info *pipe,
> >>>>> size_t len, unsigned int flags);
> >>>>> ssize_t fuse_passthrough_mmap(struct file *file, struct vm_area_struct *vma);
> >>>>>
> >>>>> +#ifdef CONFIG_SYSCTL
> >>>>> +int fuse_sysctl_register(void);
> >>>>> +void fuse_sysctl_unregister(void);
> >>>>> +#else
> >>>>> +static inline int fuse_sysctl_register(void) { return 0; }
> >>>>> +static inline void fuse_sysctl_unregister(void) { return; }
> >>>>> +#endif /* CONFIG_SYSCTL */
> >>>>> +
> >>>>> #endif /* _FS_FUSE_I_H */
> >>>>> diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
> >>>>> index 9e69006fc026..cf333448f2d3 100644
> >>>>> --- a/fs/fuse/inode.c
> >>>>> +++ b/fs/fuse/inode.c
> >>>>> @@ -35,6 +35,10 @@ DEFINE_MUTEX(fuse_mutex);
> >>>>>
> >>>>> static int set_global_limit(const char *val, const struct kernel_param *kp);
> >>>>>
> >>>>> +/* default is no timeout */
> >>>>> +u32 fuse_default_req_timeout = 0;
> >>>>> +u32 fuse_max_req_timeout = 0;
> >>>>> +
> >>>>> unsigned max_user_bgreq;
> >>>>> module_param_call(max_user_bgreq, set_global_limit, param_get_uint,
> >>>>> &max_user_bgreq, 0644);
> >>>>> @@ -1678,6 +1682,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx)
> >>>>> struct fuse_conn *fc = fm->fc;
> >>>>> struct inode *root;
> >>>>> struct dentry *root_dentry;
> >>>>> + u32 req_timeout;
> >>>>> int err;
> >>>>>
> >>>>> err = -EINVAL;
> >>>>> @@ -1730,10 +1735,16 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx)
> >>>>> fc->group_id = ctx->group_id;
> >>>>> fc->legacy_opts_show = ctx->legacy_opts_show;
> >>>>> fc->max_read = max_t(unsigned int, 4096, ctx->max_read);
> >>>>> - fc->req_timeout = ctx->req_timeout * HZ;
> >>>>> fc->destroy = ctx->destroy;
> >>>>> fc->no_control = ctx->no_control;
> >>>>> fc->no_force_umount = ctx->no_force_umount;
> >>>>> + req_timeout = ctx->req_timeout ?: fuse_default_req_timeout;
> >>>>> + if (!fuse_max_req_timeout)
> >>>>> + fc->req_timeout = req_timeout * HZ;
> >>>>> + else if (!req_timeout)
> >>>>> + fc->req_timeout = fuse_max_req_timeout * HZ;
> >>>>
> >>>> So if fuse_max_req_timeout is non-zero and req_timeout is zero (either
> >>>> because of 0 fuse_default_req_timeout, or explicit "-o request_timeout =
> >>>> 0" mount option), the final request timeout is exactly
> >>>> fuse_max_req_timeout, which is unexpected as I think 0
> >>>> fuse_default_req_timeout, or "-o request_timeout=0" shall indicate no
> >>>> timeout.
> >>>
> >>> fuse_max_req_timeout takes precedence over fuse_default_req_timeout
> >>> (eg if the system administrator wants to enforce a max limit on fuse
> >>> timeouts, that is imposed even if a specific fuse server didn't
> >>> indicate a timeout or indicated no timeout). Sorry, that wasn't made
> >>> clear in the documentation. I'll add that in for v5.
> >>
> >> OK that is quite confusing. If the system admin wants to enforce a
> >> timeout, then a non-zero fuse_default_req_timeout is adequate. What's
> >> the case where fuse_default_req_timeout must be 0, and the aystem admin
> >> has to impose the enforced timeout through fuse_max_req_timeout?
> >>
> >> IMHO the semantics of fuse_max_req_timeout is not straightforward and
> >> can be confusing if it implies an enforced timeout when no timeout is
> >> specified, while at the same time it also imposes a maximum limitation
> >> when timeout is specified.
> >
> > In my point of view, max_req_timeout is the ultimate safeguard the
> > administrator can set to enforce a timeout on all fuse requests on the
> > system (eg to mitigate rogue servers). When this is set, this
> > guarantees that absolutely no request will take longer than
> > max_req_timeout for the server to respond.
> >
> > My understanding of /proc/sys sysctls is that ACLs can be used to
> > grant certain users/groups write permission for specific sysctl
> > parameters. So if a user wants to enforce a default request timeout,
> > they can set that. If that timeout is shorter than what the max
> > request timeout has been set to, then the request should time out
> > earlier according to that desired default timeout. But if it's greater
> > than what the max request timeout allows, then the max request timeout
> > limits the timeout on the request (the max request timeout is the
> > absolute upper bound on how long a request reply can take). It doesn't
> > matter if the user set no timeout as the default req timeout - what
> > matters is that there is a max req timeout on the system, and that
> > takes precedence for enforcing how long request replies can take.
> >
>
> Sorry for the late reply, just back from vacation these days.
>
> Anyway, if max_req_timeout enforces a maximum timeout no matter whether
> the fuse server explicitly specifies a timeout or not, then the
> semantics of fuse_default_req_timeout seems a little bit overlapped with
> max_req_timeout, right? The only place where fuse_default_req_timeout
> plays a role is when ctx->req_timeout and fuse_max_req_timeout are both
> zero, in which case we can get the same effect if we eliminate
> fuse_default_req_timeout and configure a non-zero fuse_max_req_timeout.
>
The behavior would not be the same if we eliminated
fuse_default_request_timeout and configured a non-zero
fuse_max_request_timeout instead. For example, say we want a default
timeout of 10 secs. With fuse_default_request_timeout set to 10 secs,
if a server specifies a timeout that is 15 secs, that is perfectly ok.
If we get rid of fuse_default_request_timeout and just use
fuse_max_request_timeout of 10 secs, that will limit servers to 10
secs even if they specified 15 secs.
Thanks,
Joanne
> --
> Thanks,
> Jingbo
