Update to this: The problem was resolved when I explicitly mounted nfsvers=3 . I may come back to this to see if there's a reason why it's behaving like that with nfs4, but I'll need to deal with fallout for a while first. On Mon, Sep 6, 2021 at 2:37 PM John Cholewa <jcholewa@xxxxxxxxx> wrote: > > My distributed volume had an issue on Friday which required a reboot > of the primary node. After this, I'm having a strange issue: When I > have the volume mounted via ganesha-nfs, using either the primary node > itself or a random workstation on the network, I'm seeing files from > both volumes, but I'm not seeing any directories at all. It's just a > listing of the files. But I *can* list the contents of a directory if > I know it exists. Similarly, that will show the files (in both nodes) > of that directory, but it will show no subdirectories. Example: > > $ ls -F /mnt > flintstone/ > > $ ls -F /mnt/flintstone > test test1 test2 test3 > > $ ls -F /mnt/flintstone/wilma > file1 file2 file3 > > I've tried restarting glusterd on both nodes and rebooting the other > node as well. Mount options in fstab are defaults,_netdev,nofail. I > tried temporarily disabling the firewall in case that was a > contributing factor. > > This has been working pretty well for over two years, and it's > survived system updates and reboots on the nodes, and there hasn't > been a recent software update that would have triggered this. The data > itself appears to be fine. 'gluster peer status' on each node shows > that the other is connected. > > What's a good way to further troubleshoot this or to tell gluster to > figure itself out? Would "gluster volume reset" bring the > configuration to its original state without damaging the data in the > bricks? Is there something I should look out for in the logs that > might give a clue? > > Outputs: > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 18.04.4 LTS > Release: 18.04 > Codename: bionic > > > # gluster --version > glusterfs 7.5 > Repository revision: git://git.gluster.org/glusterfs.git > Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> > GlusterFS comes with ABSOLUTELY NO WARRANTY. > It is licensed to you under your choice of the GNU Lesser > General Public License, version 3 or any later version (LGPLv3 > or later), or the GNU General Public License, version 2 (GPLv2), > in all cases as published by the Free Software Foundation. > > > # gluster volume status > Status of volume: gv0 > Gluster process TCP Port RDMA Port Online Pid > ------------------------------------------------------------------------------ > Brick yuzz:/gfs/brick1/gv0 N/A N/A Y 2909 > Brick wum:/gfs/brick1/gv0 49152 0 Y 2885 > > Task Status of Volume gv0 > ------------------------------------------------------------------------------ > There are no active volume tasks > > > # gluster volume info > Volume Name: gv0 > Type: Distribute > Volume ID: dcfdeed9-8fe9-4047-b18a-1a908f003d7f > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 > Transport-type: tcp > Bricks: > Brick1: yuzz:/gfs/brick1/gv0 > Brick2: wum:/gfs/brick1/gv0 > Options Reconfigured: > nfs.disable: on > transport.address-family: inet > features.cache-invalidation: on > cluster.readdir-optimize: off > performance.parallel-readdir: off > performance.cache-size: 8GB > network.inode-lru-limit: 1000000 > performance.nfs.stat-prefetch: off > > > # gluster pool list > UUID Hostname State > 4b84240e-e73a-46da-9271-72f6001a8e18 wum Connected > 7de76707-cd99-4916-9c6b-ac6f26bda373 localhost Connected > > > Output of gluster get-state: > >>>>> > [Global] > MYUUID: 7de76707-cd99-4916-9c6b-ac6f26bda373 > op-version: 31302 > > [Global options] > > [Peers] > Peer1.primary_hostname: wum > Peer1.uuid: 4b84240e-e73a-46da-9271-72f6001a8e18 > Peer1.state: Peer in Cluster > Peer1.connected: Connected > Peer1.othernames: > > [Volumes] > Volume1.name: gv0 > Volume1.id: dcfdeed9-8fe9-4047-b18a-1a908f003d7f > Volume1.type: Distribute > Volume1.transport_type: tcp > Volume1.status: Started > Volume1.brickcount: 2 > Volume1.Brick1.path: yuzz:/gfs/brick1/gv0 > Volume1.Brick1.hostname: yuzz > Volume1.Brick1.port: 0 > Volume1.Brick1.rdma_port: 0 > Volume1.Brick1.status: Started > Volume1.Brick1.spacefree: 72715274395648Bytes > Volume1.Brick1.spacetotal: 196003244277760Bytes > Volume1.Brick2.path: wum:/gfs/brick1/gv0 > Volume1.Brick2.hostname: wum > Volume1.snap_count: 0 > Volume1.stripe_count: 1 > Volume1.replica_count: 1 > Volume1.subvol_count: 2 > Volume1.arbiter_count: 0 > Volume1.disperse_count: 0 > Volume1.redundancy_count: 0 > Volume1.quorum_status: not_applicable > Volume1.snapd_svc.online_status: Offline > Volume1.snapd_svc.inited: True > Volume1.rebalance.id: 00000000-0000-0000-0000-000000000000 > Volume1.rebalance.status: not_started > Volume1.rebalance.failures: 0 > Volume1.rebalance.skipped: 0 > Volume1.rebalance.lookedup: 0 > Volume1.rebalance.files: 0 > Volume1.rebalance.data: 0Bytes > Volume1.time_left: 0 > Volume1.gsync_count: 0 > Volume1.options.nfs.disable: on > Volume1.options.transport.address-family: inet > Volume1.options.features.cache-invalidation: on > Volume1.options.cluster.readdir-optimize: off > Volume1.options.performance.parallel-readdir: off > Volume1.options.performance.cache-size: 8GB > Volume1.options.network.inode-lru-limit: 1000000 > Volume1.options.performance.nfs.stat-prefetch: off > > > [Services] > svc1.name: glustershd > svc1.online_status: Offline > > svc2.name: nfs > svc2.online_status: Offline > > svc3.name: bitd > svc3.online_status: Offline > > svc4.name: scrub > svc4.online_status: Offline > > svc5.name: quotad > svc5.online_status: Offline > > > [Misc] > Base port: 49152 > Last allocated port: 49152 > <<<<< ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
