The issue is not with glusterd restart. We need to reproduce from beginning and add-bricks to check df -h values.
I suggest not to try on the production environment. if you have any other machines, please let me know.
On Fri, May 29, 2020 at 1:37 PM Petr Certik <petr@xxxxxxxxx> wrote:
If you mean the issue during node restart, then yes, I think I could
reproduce that with a custom build. It's a production system, though,
so I'll need to be extremely careful.
We're using debian glusterfs-server 7.3-1 amd64, can you provide a
custom glusterd binary based off of that version?
Cheers,
Petr
On Fri, May 29, 2020 at 9:09 AM Sanju Rakonde <srakonde@xxxxxxxxxx> wrote:
>
> Surprising! Will you be able to reproduce the issue and share the logs if I provide a custom build with more logs?
>
> On Thu, May 28, 2020 at 1:35 PM Petr Certik <petr@xxxxxxxxx> wrote:
>>
>> Thanks for your help! Much appreciated.
>>
>> The fsid is the same for all bricks:
>>
>> imagegluster1:
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data2-brick:brick-fsid=2065
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data-brick:brick-fsid=2065
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data-brick:brick-fsid=0
>>
>> imagegluster2:
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data2-brick:brick-fsid=2065
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data-brick:brick-fsid=2065
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data-brick:brick-fsid=0
>>
>> imagegluster3:
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster1:-data-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data2-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster2:-data-brick:brick-fsid=0
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data2-brick:brick-fsid=2065
>> /var/lib/glusterd/vols/gv0/bricks/imagegluster3:-data-brick:brick-fsid=2065
>>
>>
>> I already did try restarting the glusterd nodes with no effect, but
>> that was before the upgrades of client versions.
>>
>> Running the "volume set" command did not seem to work either, the
>> shared-brick-counts are still the same (2).
>>
>> However, when restarting a node, I do get an error and a few warnings
>> in the log: https://pastebin.com/tqq1FCwZ
>>
>>
>>
>> On Wed, May 27, 2020 at 3:14 PM Sanju Rakonde <srakonde@xxxxxxxxxx> wrote:
>> >
>> > The shared-brick-count value indicates the number of bricks sharing a file-system. In your case, it should be one, as all the bricks are from different mount points. Can you please share the values of brick-fsid?
>> >
>> > grep "brick-fsid" /var/lib/glusterd/vols/<volname>/bricks/
>> >
>> > I tried reproducing this issue in fedora vm's but couldn't hit this. we are seeing this issue on and off but are unable to reproduce in-house. If you see any error messages in glusterd.log please share the log too.
>> >
>> > Work-around to come out from this situation:
>> > 1. Restarting the glusterd service on all nodes:
>> > # systemctl restart glusterd
>> >
>> > 2. Run set volume command to update vol file:
>> > # gluster v set <VOLNAME> min-free-disk 11%
>> >
>> > On Wed, May 27, 2020 at 5:24 PM Petr Certik <petr@xxxxxxxxx> wrote:
>> >>
>> >> As far as I remember, there was no version update on the server. It
>> >> was definitely installed as version 7.
>> >>
>> >> Shared bricks:
>> >>
>> >> Server 1:
>> >>
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data2-brick.vol:
>> >> option shared-brick-count 2
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data-brick.vol: option
>> >> shared-brick-count 2
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data-brick.vol: option
>> >> shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data-brick.vol: option
>> >> shared-brick-count 0
>> >>
>> >> Server 2:
>> >>
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data-brick.vol: option
>> >> shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data2-brick.vol:
>> >> option shared-brick-count 2
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data-brick.vol: option
>> >> shared-brick-count 2
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data-brick.vol: option
>> >> shared-brick-count 0
>> >>
>> >> Server 3:
>> >>
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster1.data-brick.vol: option
>> >> shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data2-brick.vol:
>> >> option shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster2.data-brick.vol: option
>> >> shared-brick-count 0
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data2-brick.vol:
>> >> option shared-brick-count 2
>> >> /var/lib/glusterd/vols/gv0/gv0.imagegluster3.data-brick.vol: option
>> >> shared-brick-count 2
>> >>
>> >> On Wed, May 27, 2020 at 1:36 PM Sanju Rakonde <srakonde@xxxxxxxxxx> wrote:
>> >> >
>> >> > Hi Petr,
>> >> >
>> >> > what was the server version before upgrading to 7.2?
>> >> >
>> >> > Can you please share the shared-brick-count values from brick volfiles from all the nodes?
>> >> > grep shared-brick-count /var/lib/glusterd/vols/<volume_name>/*
>> >> >
>> >> > On Wed, May 27, 2020 at 2:31 PM Petr Certik <petr@xxxxxxxxx> wrote:
>> >> >>
>> >> >> Hi everyone,
>> >> >>
>> >> >> we've been running a replicated volume for a while, with three ~1 TB
>> >> >> bricks. Recently we've added three more same-sized bricks, making it a
>> >> >> 2 x 3 distributed replicated volume. However, even after rebalance,
>> >> >> the `df` command on a client shows the correct used/size percentage,
>> >> >> but wrong absolute sizes. The size still shows up as ~1 TB while in
>> >> >> reality it should be around 2 TB, and both "used" and "available"
>> >> >> reported sizes are about half of what they should be. The clients were
>> >> >> an old version (5.5), but even after upgrade to 7.2 and remount, the
>> >> >> reported sizes are still wrong. There are no heal entries. What can I
>> >> >> do to fix this?
>> >> >>
>> >> >> OS: debian buster everywhere
>> >> >> Server version: 7.3-1, opversion: 70200
>> >> >> Client versions: 5.5-3, 7.6-1, opversions: 50400, 70200
>> >> >>
>> >> >>
>> >> >> root@imagegluster1:~# gluster volume info gv0
>> >> >> Volume Name: gv0
>> >> >> Type: Distributed-Replicate
>> >> >> Volume ID: 5505d350-9b61-4056-9054-de9dfb58eab7
>> >> >> Status: Started
>> >> >> Snapshot Count: 0
>> >> >> Number of Bricks: 2 x 3 = 6
>> >> >> Transport-type: tcp
>> >> >> Bricks:
>> >> >> Brick1: imagegluster1:/data/brick
>> >> >> Brick2: imagegluster2:/data/brick
>> >> >> Brick3: imagegluster3:/data/brick
>> >> >> Brick4: imagegluster1:/data2/brick
>> >> >> Brick5: imagegluster2:/data2/brick
>> >> >> Brick6: imagegluster3:/data2/brick
>> >> >> Options Reconfigured:
>> >> >> features.cache-invalidation: on
>> >> >> transport.address-family: inet
>> >> >> storage.fips-mode-rchecksum: on
>> >> >> nfs.disable: on
>> >> >> performance.client-io-threads: off
>> >> >>
>> >> >>
>> >> >> root@imagegluster1:~# df -h
>> >> >> Filesystem Size Used Avail Use% Mounted on
>> >> >> ...
>> >> >> /dev/sdb1 894G 470G 425G 53% /data2
>> >> >> /dev/sdc1 894G 469G 426G 53% /data
>> >> >>
>> >> >>
>> >> >> root@any-of-the-clients:~# df -h
>> >> >> Filesystem Size Used Avail Use% Mounted on
>> >> >> ...
>> >> >> imagegluster:/gv0 894G 478G 416G 54% /mnt/gluster
>> >> >>
>> >> >>
>> >> >> Let me know if there's any other info I can provide about our setup.
>> >> >>
>> >> >> Cheers,
>> >> >> Petr Certik
>> >> >> ________
>> >> >>
>> >> >>
>> >> >>
>> >> >> Community Meeting Calendar:
>> >> >>
>> >> >> Schedule -
>> >> >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >> >> Bridge: https://bluejeans.com/441850968
>> >> >>
>> >> >> Gluster-users mailing list
>> >> >> Gluster-users@xxxxxxxxxxx
>> >> >> https://lists.gluster.org/mailman/listinfo/gluster-users
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > Thanks,
>> >> > Sanju
>> >>
>> >
>> >
>> > --
>> > Thanks,
>> > Sanju
>>
>
>
> --
> Thanks,
> Sanju
Thanks,
Sanju________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
