On Aug 01 2023, Miklos Szeredi <miklos@xxxxxxxxxx> wrote: > On Tue, 1 Aug 2023 at 16:40, Nikolaus Rath <nikolaus@xxxxxxxx> wrote: >> >> On Tue, 1 Aug 2023, at 13:53, Miklos Szeredi via fuse-devel wrote: >> > Here's one with the virtual env and the correct head: >> > >> > root@kvm:~/s3ql# git log -1 --pretty="%h %s" >> > 3d35f18543d9 Reproducer for notify_delete issue. To confirm: >> > root@kvm:~/s3ql# ~/s3ql-python-env/bin/python bin/s3qlrm mnt/test >> > WARNING: Received unknown command via control inode >> > ERROR: Uncaught top-level exception: >> > Traceback (most recent call last): >> > File "/root/s3ql/bin/s3qlrm", line 21, in <module> >> > s3ql.remove.main(sys.argv[1:]) >> > File "/root/s3ql/src/s3ql/remove.py", line 72, in main >> > pyfuse3.setxattr(ctrlfile, 'rmtree', cmd) >> > File "src/pyfuse3.pyx", line 629, in pyfuse3.setxattr >> > OSError: [Errno 22] Invalid argument: 'mnt/test/.__s3ql__ctrl__' >> >> This is odd. I have never heard of anyone having this problem before and it also works fine in the CI. >> >> I apologize that this is taking so much of your time. >> >> I have changed the code a bit to print out what exactly it is receiving: >> https://github.com/s3ql/s3ql/commit/eb31f7bff4bd985d68fa20c793c2f2edf5db61a5 >> >> Would you mind updating your branch and trying again? (You'll need to fetch and reset, >> since I rebased on top of current master just to be sure). >> >> I can still reproduce this every time (without any other error): >> >> $ mkdir bucket >> $ bin/mkfs.s3ql --plain local://bucket >> Before using S3QL, make sure to read the user's guide, especially >> the 'Important Rules to Avoid Losing Data' section. >> Creating metadata tables... >> Uploading metadata... >> Uploading metadata... >> Uploaded 1 out of ~1 dirty blocks (100%) >> Calculating metadata checksum... >> $ mkdir mnt >> $ bin/mount.s3ql --fg local://bucket mnt & >> Using 10 upload threads. >> Autodetected 1048514 file descriptors available for cache entries >> Using cached metadata. >> Setting cache size to 315297 MB >> Mounting local:///home/nikratio/in-progress/s3ql/bucket/ at /home/nikratio/in-progress/s3ql/mnt... >> >> $ md mnt/test; echo foo > mnt/test/bar >> $ bin/s3qlrm mnt/test >> fuse: writing device: Directory not empty >> ERROR: Failed to submit invalidate_entry request for parent inode 1, name b'test' >> Traceback (most recent call last): >> File "src/internal.pxi", line 125, in pyfuse3._notify_loop >> File "src/pyfuse3.pyx", line 915, in pyfuse3.invalidate_entry >> OSError: [Errno 39] fuse_lowlevel_notify_delete returned: Directory not empty >> >> nikratio@vostro ~/i/s3ql (notify_delete_bug)> > > WARNING: Received unknown command via control inode: b"1, b'test')" > ERROR: Uncaught top-level exception: > Traceback (most recent call last): > File "/root/s3ql/bin/s3qlrm", line 21, in <module> > s3ql.remove.main(sys.argv[1:]) > File "/root/s3ql/src/s3ql/remove.py", line 74, in main > pyfuse3.setxattr(ctrlfile, 'rmtree', cmd) > File "src/pyfuse3.pyx", line 629, in pyfuse3.setxattr > OSError: [Errno 22] Invalid argument: 'mnt/test/.__s3ql__ctrl__' Thanks! It looks like the extended attribute name and value that S3QL receives from libfuse is corrupted. What reaches S3QL as the xattr name is actually the truncated xattr value (leading parenthesis is missing). Is it possible that you are running into a variant of https://github.com/libfuse/libfuse/issues/730? This was fixed in libfuse 3.14.1 and introduced in 3.13.0. Best, -Nikolaus
