Hi Mohit, strange, after the latest kernel update (5.10.0-22-amd64 #1 Debian 5.10.178-3) and with gluster 10.4 running, i got this message again today. Result: for a certain time period e.g. 'df -h' hangs. And as gluster is the only thing (at least that i'm aware of) that uses FUSE i thought there might... ;-) Maybe i'll have to check if there's anything that can be done on kernel/system side. Searching... :-) Thx, Hubert Am Di., 2. Mai 2023 um 14:28 Uhr schrieb Mohit Agrawal <moagrawa@xxxxxxxxxx>: > > I don't think the issue is on gluster side, it seems the issue is on kernel side (possible deadlock in fuse_reverse_inval_entry) > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bda9a71980e083699a0360963c0135657b73f47a > > On Tue, May 2, 2023 at 5:48 PM Hu Bert <revirii@xxxxxxxxxxxxxx> wrote: >> >> Good morning, >> >> we've recently had some strange message in /var/log/syslog. >> >> System: >> debian bullseye, kernel 5.10.0-21-amd64 and 5.10.0-22-amd64 >> gluster 10.3 >> >> The message look like: >> >> Apr 27 13:30:18 piggy kernel: [24287.715229] INFO: task >> glfs_fusenoti:2787 blocked for more than 120 seconds. >> Apr 27 13:30:18 piggy kernel: [24287.715327] Not tainted >> 5.10.0-22-amd64 #1 Debian 5.10.178-3 >> Apr 27 13:30:18 piggy kernel: [24287.715419] "echo 0 > >> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Apr 27 13:30:18 piggy kernel: [24287.715575] task:glfs_fusenoti >> state:D stack: 0 pid: 2787 ppid: 1 flags:0x00000000 >> Apr 27 13:30:18 piggy kernel: [24287.715734] Call Trace: >> Apr 27 13:30:18 piggy kernel: [24287.715847] __schedule+0x282/0x870 >> Apr 27 13:30:18 piggy kernel: [24287.715959] schedule+0x46/0xb0 >> Apr 27 13:30:18 piggy kernel: [24287.716073] >> rwsem_down_write_slowpath+0x257/0x4d0 >> Apr 27 13:30:18 piggy kernel: [24287.716194] >> fuse_reverse_inval_entry+0x3b/0x1e0 [fuse] >> >> etc. Full excerpt here: https://pastebin.com/6gDHgh16 >> >> This may cause our application to "hang". In the latest release i read >> something about fuse: >> https://docs.gluster.org/en/latest/release-notes/10.4/ >> "Fix fuse concurrency problems" >> >> but i checked the tickets and wasn't able to find something. >> >> However, i upgraded gluster client 10.3 -> 10.4, hoping that this >> fixes the hang-issue. >> >> Has anyone seen these messages before? >> >> >> Best regards, >> Hubert >> ------- >> >> Community Meeting Calendar: >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://meet.google.com/cpu-eiue-hvk >> >> Gluster-devel mailing list >> Gluster-devel@xxxxxxxxxxx >> https://lists.gluster.org/mailman/listinfo/gluster-devel >> ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
