On Tue, Jul 30, 2024 at 8:28 AM Joanne Koong <joannelkoong@xxxxxxxxx> wrote: > > There are situations where fuse servers can become unresponsive or take > too long to reply to a request. Currently there is no upper bound on > how long a request may take, which may be frustrating to users who get > stuck waiting for a request to complete. > > This patchset adds a timeout option for requests and two dynamically > configurable fuse sysctls "default_request_timeout" and "max_request_timeout" > for controlling/enforcing timeout behavior system-wide. > > Existing fuse servers will not be affected unless they explicitly opt into the > timeout. > > v1: https://lore.kernel.org/linux-fsdevel/20240717213458.1613347-1-joannelkoong@xxxxxxxxx/ > Changes from v1: > - Add timeout for background requests > - Handle resend race condition > - Add sysctls > > Joanne Koong (2): > fuse: add optional kernel-enforced timeout for requests > fuse: add default_request_timeout and max_request_timeout sysctls > > Documentation/admin-guide/sysctl/fs.rst | 17 +++ > fs/fuse/Makefile | 2 +- > fs/fuse/dev.c | 187 +++++++++++++++++++++++- > fs/fuse/fuse_i.h | 30 ++++ > fs/fuse/inode.c | 24 +++ > fs/fuse/sysctl.c | 42 ++++++ > 6 files changed, 293 insertions(+), 9 deletions(-) > create mode 100644 fs/fuse/sysctl.c > > -- > 2.43.0 > Hello Joanne, Thanks for your update. I have tested your patches using my test case, which is similar to the hello-fuse [0] example, with an additional change as follows: @@ -125,6 +125,8 @@ static int hello_read(const char *path, char *buf, size_t size, off_t offset, } else size = 0; + // TO trigger timeout + sleep(60); return size; } [0] https://github.com/libfuse/libfuse/blob/master/example/hello.c However, it triggered a crash with the following setup: 1. Set FUSE timeout: sysctl -w fs.fuse.default_request_timeout=10 sysctl -w fs.fuse.max_request_timeout = 20 2. Start FUSE daemon: ./hello /tmp/fuse 3. Read from FUSE: cat /tmp/fuse/hello 4. Kill the process within 10 seconds (to avoid the timeout being triggered). Then the crash will be triggered. The crash details are as follows: [ 270.729966] CPU: 37 PID: 0 Comm: swapper/37 Kdump: loaded Not tainted 6.10.0+ #30 [ 270.731658] RIP: 0010:__run_timers+0x27e/0x360 [ 270.732129] Code: 07 48 c7 43 08 00 00 00 00 48 85 c0 74 78 4d 8b 2f 4c 89 6b 08 0f 1f 44 00 00 49 8b 45 00 49 8b 55 08 48 89 02 48 85 c0 74 04 <48> 89 50 08 4d 8b 65 18 49 c7 45 08 00 00 00 00 48 b8 22 01 00 00 [ 270.733815] RSP: 0018:ffff9c1c80d00ed8 EFLAGS: 00010086 [ 270.734347] RAX: dead000000000122 RBX: ffff8bfb7f7613c0 RCX: 0000000000000001 [ 270.735037] RDX: ffff9c1c80d00ef8 RSI: 0000000000000000 RDI: ffff8bfb7f7613e8 [ 270.735723] RBP: ffff9c1c80d00f70 R08: 00000000000000b3 R09: ffff8bfb7f761430 [ 270.736439] R10: ffffffffb0e060c0 R11: 00000000000000b1 R12: 0000000000000001 [ 270.737133] R13: ffff8bbd6591a0a0 R14: 00000000ffff8c00 R15: ffff9c1c80d00ef8 [ 270.737834] FS: 0000000000000000(0000) GS:ffff8bfb7f740000(0000) knlGS:0000000000000000 [ 270.738603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 270.739178] CR2: 00007fac75f44778 CR3: 00000001200c2006 CR4: 0000000000370ef0 [ 270.739880] Call Trace: [ 270.740211] <IRQ> [ 270.740512] ? show_regs+0x69/0x80 [ 270.740934] ? die_addr+0x38/0x90 [ 270.741340] ? exc_general_protection+0x236/0x490 [ 270.741874] ? asm_exc_general_protection+0x27/0x30 [ 270.742416] ? __run_timers+0x27e/0x360 [ 270.742872] ? __run_timers+0x1b4/0x360 [ 270.743318] ? kvm_sched_clock_read+0x11/0x20 [ 270.743821] ? sched_clock_noinstr+0x9/0x10 [ 270.744298] ? sched_clock+0x10/0x30 [ 270.744716] ? sched_clock_cpu+0x10/0x190 [ 270.745190] run_timer_softirq+0x3a/0x60 [ 270.745647] handle_softirqs+0x118/0x350 [ 270.746106] irq_exit_rcu+0x60/0x80 [ 270.746527] sysvec_apic_timer_interrupt+0x7f/0x90 [ 270.747067] </IRQ> [ 270.747374] <TASK> [ 270.747669] asm_sysvec_apic_timer_interrupt+0x1b/0x20 [ 270.748255] RIP: 0010:default_idle+0xb/0x20 [ 270.748724] Code: 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 eb 07 0f 00 2d b3 51 33 00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 [ 270.750507] RSP: 0018:ffff9c1c801e7e18 EFLAGS: 00000246 [ 270.751075] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000028c5f01d9e8e [ 270.751811] RDX: 0000000000000001 RSI: ffffffffb112e200 RDI: ffff8bfb7f77c8e0 [ 270.752529] RBP: ffff9c1c801e7e20 R08: 0000003f08a215b6 R09: 0000000000000001 [ 270.753298] R10: ffffffffb0e56080 R11: 0000000000000001 R12: 0000000000000001 [ 270.754023] R13: ffffffffb112e200 R14: ffffffffb112e280 R15: 0000000000000001 [ 270.754742] ? ct_kernel_exit.constprop.0+0x79/0x90 [ 270.755285] ? arch_cpu_idle+0x9/0x10 [ 270.755707] default_enter_idle+0x22/0x2f [ 270.756175] cpuidle_enter_state+0x88/0x430 [ 270.756648] cpuidle_enter+0x34/0x50 [ 270.757075] call_cpuidle+0x22/0x50 [ 270.757492] cpuidle_idle_call+0xd2/0x120 [ 270.757960] do_idle+0x77/0xd0 [ 270.758347] cpu_startup_entry+0x2c/0x30 [ 270.758804] start_secondary+0x117/0x140 [ 270.759260] common_startup_64+0x13e/0x141 [ 270.759721] </TASK> Please feel free to reach out if you are unable to reproduce the issue. -- Regards Yafang
