Il 24/06/21 03:45, cfeller@xxxxxxxxxxxxxx ha scritto:
Probably related to the problem I reported some time ago (apr, 22nd),
thread "quantum directories?". Got no answer :(
Hi,
I've got an interesting issue with files not being found when accessed directly.
When accessing a file like so:
#ls /mnt/path/to/some/file/file.json
ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory
I can wait, try again, and same result. I could put this in a loop with a delay between read attempts, same result.
However, if I do an "ls" of the directory first then everything works. That is:
# ls /mnt/path/to/some/file/file.json
ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory
# ls /mnt/path/to/some/file/
dir1 dir2 file1.txt file2.txt file3.txt file.json
At this point I can successfully "ls" the file:
# ls /mnt/path/to/some/file/file.json
/mnt/path/to/some/file/file.json
It is curious, but seems to be isolated to certain directories/mount points.
Looking at the mount logs there is absolutely nothing printed to the mnt log on the client with regards to the failure or the eventual success.
The configuration:
Distributed-Replicate
4 x 2 = 8
Gluster version 9.2
Clients are using native FUSE mounts (also version 9.2)
I did recently add the 4th brick, and the system is currently undergoing a rebalance (the fix-layout rebalance already completed). I am throttling the full rebalance as 'lazy'.
I looked at the bricks on the server and the files still exists on the older bricks, so I don't think the files in question got rebalanced (yet).
The system load isn't too high, and this doesn't seem to be a consistent error as I don't have this issue in other directories, but I consistently have the issue on this directory as well as a few others.
Adding some additional background/history:
Cluster was setup a couple of years ago on gluster 6.x.
Did an expansion (two replica pairs to three replica pairs) and rebalance while on 6.x, no issues.
Upgraded to gluster 7.x -> 8.x -> 9.x
Ops version has been upgraded to 90000
Doing another expansion (three replica pairs to four replica pairs) and rebalance now, and seeing some interesting issues, including this one.
There is 0 entries under "heal info" or "info split-brain".
I'm just wondering if anyone on this list has seen anything similar, or has any suggestions.
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
