Yes, I compared the client count like this:
gluster volume status <vol> clients |grep -B1 connected
I ran the find command on each client before and after shutting down the problematic daemon to determine any file count differences:
find /mount/point |wc -l
After my last post I discovered that one of the clients had somehow been blocked by iptables from connecting to one of the bricks. So for an extended period any file creation from that one client was perpetuating an imbalance between bricks, causing different files to be visible for different clients. What baffles me is that gluster wouldn't automatically fix an imbalance between replicas like that.
On Fri, Nov 20, 2020, at 1:24 AM, Benedikt Kaleß wrote:
Dear James,
we have exactly the same problems.
Could you describe what you did to discover which of your bricks had the worst file count discrepancy and how you find out that all clients matched after shutting down this daemon?
Best regards
Benedikt
Am 02.11.20 um 17:30 schrieb James H:I found a solution after making a discovery. I logged into the brick with the worst file count discrepancy - odroid4 - and killed the gluster daemon there. All file counts across all clients then matched. So I started the daemon and ran this command to try to fix it up:
gluster volume replace-brick gvol0 odroid4:/srv/gfs-brick/gvol0 odroid4:/srv/gfs-brick/gvol0_2 commit force
...and that fixed it. It's disconcerting that it's possible for Gluster to merrily hum along without any problems showing up in the various status summaries yet show vastly different directory listings to different clients. Is this a known problem or shall I open a bug report? Are there any particular error logs I should monitor to be alerted to this bad state?
On Thu, Oct 29, 2020 at 8:39 PM James H <stormdig@xxxxxxxxx> wrote:Hi folks, I'm struggling to find a solution to missing files on FUSE mounts. Which files are missing is different on different clients. I can stat or ls the missing files directly when called by filename but listing directories won't show them.So far I've:
- verified heal info shows no files in need of healing and no split brain condition
- verified the same number of clients are connected to each brick
- verified the file counts on the bricks match
- upgraded Gluster server and clients from 3.x to 6.x and 7.x
- run a stat on all files
- run a heal full
- rebooted / remounted FUSE clients
File count from running a 'find' command on FUSE mounts on the bricks themselves. These counts should all be the same:38823 fuse-odroid1-share238823 fuse-odroid2-share260962 fuse-odroid3-share27202 fuse-odroid4-share2...and a FUSE mount on a seperate server:38823 fuse-phn2dsm-share2File count from running a 'find' command on brick directories themselves::43382 brick-odroid1-share243382 brick-odroid2-share243382 brick-arbiter-odroid3-share223075 brick-odroid3-share223075 brick-odroid4-share223075 brick-arbiter-odroid2-share2Here's some info about the setup:# gluster --version | head -1; cat /etc/lsb-release; uname -rglusterfs 7.8DISTRIB_ID=UbuntuDISTRIB_RELEASE=18.04DISTRIB_CODENAME=bionicDISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"4.14.157-171# gluster volume infoVolume Name: gvol0Type: Distributed-ReplicateVolume ID: 57e3a085-5fb7-417d-a71a-fed5cd0ae2d9Status: StartedSnapshot Count: 0Number of Bricks: 2 x (2 + 1) = 6Transport-type: tcpBricks:Brick1: odroid1:/srv/gfs-brick/gvol0Brick2: odroid2:/srv/gfs-brick/gvol0Brick3: odroid3:/srv/gfs-brick/gvol0-arbiter2 (arbiter)Brick4: odroid3:/srv/gfs-brick/gvol0_2Brick5: odroid4:/srv/gfs-brick/gvol0Brick6: odroid2:/srv/gfs-brick/gvol0-arbiter2 (arbiter)Options Reconfigured:cluster.self-heal-daemon: enableperformance.readdir-ahead: yesperformance.cache-invalidation: onperformance.stat-prefetch: onperformance.quick-read: oncluster.shd-max-threads: 4performance.parallel-readdir: oncluster.server-quorum-type: serverserver.event-threads: 4client.event-threads: 4performance.nl-cache-timeout: 600performance.nl-cache: onnetwork.inode-lru-limit: 200000performance.md-cache-timeout: 600performance.cache-samba-metadata: onfeatures.cache-invalidation-timeout: 600features.cache-invalidation: onstorage.fips-mode-rchecksum: onperformance.client-io-threads: offnfs.disable: ontransport.address-family: inetfeatures.bitrot: onfeatures.scrub: Activefeatures.scrub-throttle: lazyfeatures.scrub-freq: dailycluster.min-free-disk: 10%# gluster volume status gvol0 detailStatus of volume: gvol0------------------------------------------------------------------------------Brick : Brick odroid1:/srv/gfs-brick/gvol0TCP Port : 49152RDMA Port : 0Online : YPid : 702File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 983.4GBTotal Disk Space : 5.5TBInode Count : 586052224Free Inodes : 585835873------------------------------------------------------------------------------Brick : Brick odroid2:/srv/gfs-brick/gvol0TCP Port : 49152RDMA Port : 0Online : YPid : 30206File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 983.3GBTotal Disk Space : 5.5TBInode Count : 586052224Free Inodes : 585711242------------------------------------------------------------------------------Brick : Brick odroid3:/srv/gfs-brick/gvol0-arbiter2TCP Port : 49152RDMA Port : 0Online : YPid : 32449File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 1.4TBTotal Disk Space : 2.7TBInode Count : 293026624Free Inodes : 292378835------------------------------------------------------------------------------Brick : Brick odroid3:/srv/gfs-brick/gvol0_2TCP Port : 49153RDMA Port : 0Online : YPid : 32474File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 1.4TBTotal Disk Space : 2.7TBInode Count : 293026624Free Inodes : 292378835------------------------------------------------------------------------------Brick : Brick odroid4:/srv/gfs-brick/gvol0TCP Port : 49152RDMA Port : 0Online : YPid : 23138File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 1.4TBTotal Disk Space : 2.7TBInode Count : 293026624Free Inodes : 292891910------------------------------------------------------------------------------Brick : Brick odroid2:/srv/gfs-brick/gvol0-arbiter2TCP Port : 49153RDMA Port : 0Online : YPid : 30231File System : xfsDevice : /dev/sdaMount Options : rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquotaInode Size : 512Disk Space Free : 983.3GBTotal Disk Space : 5.5TBInode Count : 586052224Free Inodes : 585711242________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users-- forumZFD Entschieden für Frieden|Committed to Peace Benedikt Kaleß Leiter Team IT|Head team IT Forum Ziviler Friedensdienst e.V.|Forum Civil Peace Service Am Kölner Brett 8 | 50825 Köln | Germany Tel 0221 91273233 | Fax 0221 91273299 | http://www.forumZFD.de Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board: Oliver Knabe (Vorsitz|Chair), Sonja Wiekenberg-Mlalandle, Alexander Mauz VR 17651 Amtsgericht Köln Spenden|Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
