> Can you provide the volume info. > Also check the logs for the rebalance. There should be some valuable info on what is going on. Hello thank you for replying. The re-balance log is between 20M and 40M on all 9 nodes. Even xz-compressing one gets us to 1.8M which I'm not sure I can attach here. However, looking at one, I saw most lines were the same. So I tried removing the repeating normal lines like this: # cat cm_shared-rebalance.log | egrep -v "completed migration of|migrate data called on|attempting to move" > leader1-cm_shared-rebalance.log-filtered # xz -9 leader1-cm_shared-rebalance.log-filtered So I have attached the above and the volume info. I'm not sure what I'm looking for exactly and I know the above is just one of the 9 leaders. It's almost like the gluster NFS server was losing track of the moves in terms of serving the clients. Thanks again !!! Attached some details! Happy to attach more! Finally a problem I can reproduce at will in the lab :) :) PS: I can make stuff available on an ftp server too. On Thu, Jan 30, 2020 at 05:50:30PM +0200, Strahil Nikolov wrote: > On January 30, 2020 2:20:34 AM GMT+02:00, Erik Jacobson <erik.jacobson@xxxxxxx> wrote: > >We are using gluster 4.1.6. We are using gluster NFS (not ganesha). > > > >Distributed/replicated with subvolume size 3 (6 total servers, 2 > >subvols). > > > >The NFS clients use this for their root filesystem. > > > >When I add 3 more gluster servers to add one more subvolume to the > >storage volumes (so now subvolume size 3, 9 total servers, 3 total > >subvolumes), the process gets started. > > > >ssh leader1 gluster volume add-brick cm_shared > >172.23.0.9://data/brick_cm_shared 172.23.0.10://data/brick_cm_shared > >172.23.0.11://data/brick_cm_shared > > > >then > > > >ssh leader1 gluster volume rebalance cm_shared start > > > >The re-balance works. 'gluster volume status' shows re-balance in > >progress. > > > >However, existing gluster-NFS clients now show missing files and I can > >no longer log into them (since NFS is their root). If you are logged > >in, > >you can find that libraries are missing and general unhappiness with > >random files now missing. > > > >Is accessing a volume that is in the process of being re-balanced not > >supported from a gluster NFS client? Or have I made an error? > > > >Thank you for any help, > > > >Erik > >________ > > > >Community Meeting Calendar: > > > >APAC Schedule - > >Every 2nd and 4th Tuesday at 11:30 AM IST > >Bridge: https://bluejeans.com/441850968 > > > >NA/EMEA Schedule - > >Every 1st and 3rd Tuesday at 01:00 PM EDT > >Bridge: https://bluejeans.com/441850968 > > > >Gluster-users mailing list > >Gluster-users@xxxxxxxxxxx > >https://lists.gluster.org/mailman/listinfo/gluster-users > > Hi Erik, > > Can you provide the volume info. > Also check the logs for the rebalance. There should be some valuable info on what is going on. > > Best Regards, > Strahil Nikolov
Attachment:
gluster-volinfo.txt.xz
Description: application/xz
Attachment:
leader1-cm_shared-rebalance.log-filtered.xz
Description: application/xz
Attachment:
volume-get-all.txt.xz
Description: application/xz
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
