Hi, I've got an interesting issue with files not being found when accessed directly. When accessing a file like so: #ls /mnt/path/to/some/file/file.json ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory I can wait, try again, and same result. I could put this in a loop with a delay between read attempts, same result. However, if I do an "ls" of the directory first then everything works. That is: # ls /mnt/path/to/some/file/file.json ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory # ls /mnt/path/to/some/file/ dir1 dir2 file1.txt file2.txt file3.txt file.json At this point I can successfully "ls" the file: # ls /mnt/path/to/some/file/file.json /mnt/path/to/some/file/file.json It is curious, but seems to be isolated to certain directories/mount points. Looking at the mount logs there is absolutely nothing printed to the mnt log on the client with regards to the failure or the eventual success. The configuration: Distributed-Replicate 4 x 2 = 8 Gluster version 9.2 Clients are using native FUSE mounts (also version 9.2) I did recently add the 4th brick, and the system is currently undergoing a rebalance (the fix-layout rebalance already completed). I am throttling the full rebalance as 'lazy'. I looked at the bricks on the server and the files still exists on the older bricks, so I don't think the files in question got rebalanced (yet). The system load isn't too high, and this doesn't seem to be a consistent error as I don't have this issue in other directories, but I consistently have the issue on this directory as well as a few others. Adding some additional background/history: Cluster was setup a couple of years ago on gluster 6.x. Did an expansion (two replica pairs to three replica pairs) and rebalance while on 6.x, no issues. Upgraded to gluster 7.x -> 8.x -> 9.x Ops version has been upgraded to 90000 Doing another expansion (three replica pairs to four replica pairs) and rebalance now, and seeing some interesting issues, including this one. There is 0 entries under "heal info" or "info split-brain". I'm just wondering if anyone on this list has seen anything similar, or has any suggestions. ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
