missing files on FUSE mounts

[James H <stormdig@xxxxxxxxx>]

Hi folks, I'm struggling to find a solution to missing files on FUSE mounts. Which files are missing is different on different clients. I can stat or ls the missing files directly when called by filename but listing directories won't show them.

So far I've:

• verified heal info shows no files in need of healing and no split brain condition
• verified the same number of clients are connected to each brick 
• verified the file counts on the bricks match
• upgraded Gluster server and clients from 3.x to 6.x and 7.x
• run a stat on all files
• run a heal full
• rebooted / remounted FUSE clients

File count from running a 'find' command on FUSE mounts on the bricks themselves. These counts should all be the same:

38823 fuse-odroid1-share2

38823 fuse-odroid2-share2

60962 fuse-odroid3-share2

7202 fuse-odroid4-share2

...and a FUSE mount on a seperate server:

38823 fuse-phn2dsm-share2  

File count from running a 'find' command on brick directories themselves::  

43382 brick-odroid1-share2

43382 brick-odroid2-share2

43382 brick-arbiter-odroid3-share2

23075 brick-odroid3-share2

23075 brick-odroid4-share2

23075 brick-arbiter-odroid2-share2

Here's some info about the setup:

# gluster --version | head -1; cat /etc/lsb-release; uname -r
glusterfs 7.8
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
4.14.157-171

# gluster volume info

Volume Name: gvol0
Type: Distributed-Replicate
Volume ID: 57e3a085-5fb7-417d-a71a-fed5cd0ae2d9
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: odroid1:/srv/gfs-brick/gvol0
Brick2: odroid2:/srv/gfs-brick/gvol0
Brick3: odroid3:/srv/gfs-brick/gvol0-arbiter2 (arbiter)
Brick4: odroid3:/srv/gfs-brick/gvol0_2
Brick5: odroid4:/srv/gfs-brick/gvol0
Brick6: odroid2:/srv/gfs-brick/gvol0-arbiter2 (arbiter)
Options Reconfigured:
cluster.self-heal-daemon: enable
performance.readdir-ahead: yes
performance.cache-invalidation: on
performance.stat-prefetch: on
performance.quick-read: on
cluster.shd-max-threads: 4
performance.parallel-readdir: on
cluster.server-quorum-type: server
server.event-threads: 4
client.event-threads: 4
performance.nl-cache-timeout: 600
performance.nl-cache: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
storage.fips-mode-rchecksum: on
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
features.bitrot: on
features.scrub: Active
features.scrub-throttle: lazy
features.scrub-freq: daily
cluster.min-free-disk: 10%

# gluster volume status gvol0 detail
Status of volume: gvol0
------------------------------------------------------------------------------
Brick                : Brick odroid1:/srv/gfs-brick/gvol0
TCP Port             : 49152
RDMA Port            : 0
Online               : Y
Pid                  : 702
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 983.4GB
Total Disk Space     : 5.5TB
Inode Count          : 586052224
Free Inodes          : 585835873
------------------------------------------------------------------------------
Brick                : Brick odroid2:/srv/gfs-brick/gvol0
TCP Port             : 49152
RDMA Port            : 0
Online               : Y
Pid                  : 30206
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 983.3GB
Total Disk Space     : 5.5TB
Inode Count          : 586052224
Free Inodes          : 585711242
------------------------------------------------------------------------------
Brick                : Brick odroid3:/srv/gfs-brick/gvol0-arbiter2
TCP Port             : 49152
RDMA Port            : 0
Online               : Y
Pid                  : 32449
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 1.4TB
Total Disk Space     : 2.7TB
Inode Count          : 293026624
Free Inodes          : 292378835
------------------------------------------------------------------------------
Brick                : Brick odroid3:/srv/gfs-brick/gvol0_2
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 32474
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 1.4TB
Total Disk Space     : 2.7TB
Inode Count          : 293026624
Free Inodes          : 292378835
------------------------------------------------------------------------------
Brick                : Brick odroid4:/srv/gfs-brick/gvol0
TCP Port             : 49152
RDMA Port            : 0
Online               : Y
Pid                  : 23138
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 1.4TB
Total Disk Space     : 2.7TB
Inode Count          : 293026624
Free Inodes          : 292891910
------------------------------------------------------------------------------
Brick                : Brick odroid2:/srv/gfs-brick/gvol0-arbiter2
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 30231
File System          : xfs
Device               : /dev/sda
Mount Options        : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota
Inode Size           : 512
Disk Space Free      : 983.3GB
Total Disk Space     : 5.5TB
Inode Count          : 586052224
Free Inodes          : 585711242

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users