On 16 April 2018 at 14:07, Raghavendra Gowdappa <rgowdapp@xxxxxxxxxx> wrote:
On Mon, Apr 16, 2018 at 1:54 PM, Niels Hendriks <niels@xxxxxxxxxx> wrote:Hi,We have a 3-node gluster setup where gluster is both the server and theclient.Every few days we have some $random file or directory that does not existaccording to the FUSE mountpoint. When we try to access the file (stat,cat, etc...) the filesystem reports that the file/directory does not exist,even though it does. When we try to create the file/directory we receivethe following error which is also logged in/var/log/glusterfs/bricks/$brick.log: [2018-04-10 12:51:26.755928] E [MSGID: 113027] [posix.c:1779:posix_mkdir]0-www-posix: mkdir of /storage/gluster/path/to/dir failed [File exists]We don't see this issue on all of the servers, but only on the servers thatdid not create the file/directory (so 2 of the 3 gluster nodes).We found that this issue does not resolve itself automatically. However,when we perform an ls command on the parent directory the issue will beresolved for the other nodes.We are running glusterfs 3.12.6 on debian 8Mount-options in /etc/fstab:/dev/storage-gluster/gluster /storage/gluster xfs rw,inode64,noatime,nouuid0 2localhost:/www /var/www glusterfsbackup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING 0 0gluster volume info wwwVolume Name: wwwType: ReplicateVolume ID: e0579d53-f671-4868-863b-ba85c4cfacb3 Status: StartedSnapshot Count: 0Number of Bricks: 1 x 3 = 3Transport-type: tcpBricks:Brick1: n01c01-gluster:/storage/gluster/www Brick2: n02c01-gluster:/storage/gluster/www Brick3: n03c01-gluster:/storage/gluster/www Options Reconfigured:performance.read-ahead: onperformance.client-io-threads: onnfs.disable: ontransport.address-family: inetperformance.md-cache-timeout: 600diagnostics.brick-log-level: WARNINGnetwork.ping-timeout: 3features.cache-invalidation: onserver.event-threads: 4performance.cache-invalidation: on performance.quick-read: onfeatures.cache-invalidation-timeout: 600 network.inode-lru-limit: 90000performance.cache-priority: *.php:3,*.temp:3,*:1performance.nl-cache: onperformance.cache-size: 1GBperformance.readdir-ahead: onperformance.write-behind: oncluster.readdir-optimize: onperformance.io-thread-count: 64client.event-threads: 4cluster.lookup-optimize: onperformance.parallel-readdir: offperformance.write-behind-window-size: 4MB performance.flush-behind: onfeatures.bitrot: onfeatures.scrub: Activeperformance.io-cache: offperformance.stat-prefetch: onWe suspected that the md-cache could be the cause, but it does have atimeout of 600 seconds so this would be strange since the issue can bepresent for hours (at which point we did an ls to fix it).Does anyone have an idea of what could be the cause of this?For files, it could be because of:* cluster.lookup-optimize set to on* datafile is present on non hashed subvol, but linkto file is absent on hashed subvol
This is a pure replicate volume:
Type: Replicate
Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
So unlikely to be a lookup-optimize problem.
I see that lookup-optimize is on. Can you get the following information from problematic file?* Name of the file* all xattrs on parent directory from all bricks* stat of file from all bricks where it is present.* all xattrs on file from all bricks where it is present.If you are seeing the problem on directory,* Name of directory* xattr of directory and its parent from all bricksregards,RaghavendraThanks!
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
