Re: Gluster 9.3 not distributing data evenly across bricks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



IIUC you have to manually run a rebalance
gluster v rebalance datavol start

Diego

Il 01/12/2023 13:37, Shreyansh Shah ha scritto:
Hi,
We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We are noticing uneven data distribution across the bricks. Some bricks are 100% utilised whereas some are < 85%. Due to this, data write randomly fails with error OSError(28, 'No space left on device') even when the volume has sufficient space on other bricks. We expect the glusterfs process to automatically sync and maintain equal data distribution across the bricks.

This is the volume info:

    root@nas-5:~# gluster v info

    Volume Name: datavol
    Type: Distribute
    Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 47
    Transport-type: tcp
    Bricks:
    Brick1: 10.132.2.101:/data/data
    Brick2: 10.132.2.101:/data1/data
    Brick3: 10.132.2.101:/data2/data
    Brick4: 10.132.2.101:/data3/data
    Brick5: 10.132.2.102:/data/data
    Brick6: 10.132.2.102:/data1/data
    Brick7: 10.132.2.102:/data2/data
    Brick8: 10.132.2.102:/data3/data
    Brick9: 10.132.2.103:/data/data
    Brick10: 10.132.2.103:/data1/data
    Brick11: 10.132.2.103:/data2/data
    Brick12: 10.132.2.103:/data3/data
    Brick13: 10.132.2.104:/data/data
    Brick14: 10.132.2.104:/data3/data
    Brick15: 10.132.2.105:/data1/data
    Brick16: 10.132.2.105:/data2/data
    Brick17: 10.132.2.106:/data/data
    Brick18: 10.132.2.106:/data1/data
    Brick19: 10.132.2.106:/data2/data
    Brick20: 10.132.2.107:/data/data
    Brick21: 10.132.2.107:/data1/data
    Brick22: 10.132.2.107:/data2/data
    Brick23: 10.132.2.108:/data/data
    Brick24: 10.132.2.108:/data1/data
    Brick25: 10.132.2.108:/data2/data
    Brick26: 10.132.2.109:/data/data
    Brick27: 10.132.2.109:/data1/data
    Brick28: 10.132.2.109:/data2/data
    Brick29: 10.132.2.110:/data/data
    Brick30: 10.132.2.110:/data1/data
    Brick31: 10.132.2.111:/data/data
    Brick32: 10.132.2.111:/data1/data
    Brick33: 10.132.2.111:/data2/data
    Brick34: 10.132.2.112:/data/data
    Brick35: 10.132.2.112:/data1/data
    Brick36: 10.132.2.112:/data2/data
    Brick37: 10.132.2.113:/data/data
    Brick38: 10.132.2.113:/data1/data
    Brick39: 10.132.2.113:/data2/data
    Brick40: 10.132.2.108:/data3/data
    Brick41: 10.132.2.107:/data3/data
    Brick42: 10.132.2.106:/data3/data
    Brick43: 10.132.2.105:/data3/data
    Brick44: 10.132.2.110:/data2/data
    Brick45: 10.132.2.105:/data/data
    Brick46: 10.132.2.104:/data1/data
    Brick47: 10.132.2.104:/data2/data
    Options Reconfigured:
    performance.client-io-threads: on
    cluster.min-free-disk: 2%
    performance.cache-refresh-timeout: 60
    client.event-threads: 4
    server.event-threads: 4
    network.ping-timeout: 90
    storage.health-check-interval: 60
    storage.health-check-timeout: 60
    performance.io-cache-size: 8GB
    storage.fips-mode-rchecksum: on
    transport.address-family: inet
    nfs.disable: on
    root@nas-5:~# glusterfs --version
    glusterfs 9.3
    Repository revision: git://git.gluster.org/glusterfs.git
    <http://git.gluster.org/glusterfs.git>
    Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/
    <https://www.gluster.org/>>
    GlusterFS comes with ABSOLUTELY NO WARRANTY.
    It is licensed to you under your choice of the GNU Lesser
    General Public License, version 3 or any later version (LGPLv3
    or later), or the GNU General Public License, version 2 (GPLv2),
    in all cases as published by the Free Software Foundation.
root@nas-5:~#


This is the disk usage distribution when the write failed:
*Node 1:*
/dev/bcache1    9.7T  8.5T  1.2T  88% /data
/dev/bcache3    3.9T  3.3T  563G  86% /data1
/dev/bcache0    3.9T  3.3T  560G  86% /data2
/dev/bcache2    3.9T  3.5T  395G  90% /data3
*Node 2:*
/dev/bcache0    9.7T  8.5T  1.2T  88% /data
/dev/bcache1    3.9T  3.5T  421G  90% /data1
/dev/bcache2    3.9T  3.6T  330G  92% /data2
/dev/bcache3    3.9T  3.4T  513G  87% /data3
*Node 3:*
/dev/bcache0    9.7T  8.8T  882G  92% /data
/dev/bcache1    3.9T  3.6T  335G  92% /data1
/dev/bcache2    3.9T  3.4T  532G  87% /data2
/dev/bcache3    3.9T  3.3T  564G  86% /data3
*Node 4:*
/dev/bcache0    9.7T  8.7T  982G  91% /data
/dev/bcache1    3.9T  3.5T  424G  90% /data1
/dev/bcache2    3.9T  3.4T  549G  87% /data2
/dev/bcache3    3.9T  3.6T  344G  92% /data3
*Node 5:*
/dev/bcache0    9.7T  8.5T  1.3T  88% /data
/dev/bcache1    3.9T  3.6T  288G  93% /data1
/dev/bcache2    3.9T  3.4T  470G  89% /data2
/dev/bcache3    9.9T  9.8T  101G 100% /data3
*Node 6:*
/dev/bcache0    9.7T  8.2T  1.5T  86% /data
/dev/bcache1    3.9T  3.4T  526G  87% /data1
/dev/bcache2    3.9T  3.5T  431G  90% /data2
/dev/bcache3    9.9T  8.9T  1.1T  90% /data3
*Node 7:*
/dev/bcache0    9.7T  8.9T  783G  93% /data
/dev/bcache1    3.9T  3.3T  561G  86% /data1
/dev/bcache2    3.9T  3.5T  360G  91% /data2
/dev/bcache3    9.9T  8.7T  1.2T  89% /data3
*Node 8:*
/dev/bcache0    9.7T  8.7T  994G  90% /data
/dev/bcache1    3.9T  3.3T  645G  84% /data1
/dev/bcache2    3.9T  3.4T  519G  87% /data2
/dev/bcache3    9.9T  9.0T  868G  92% /data3
*Node 9:*
/dev/bcache0     10T  8.6T  1.4T  87% /data
/dev/bcache1    8.0T  6.7T  1.4T  84% /data1
/dev/bcache2    8.0T  6.8T  1.3T  85% /data2
*Node 10:*
/dev/bcache0     10T  8.8T  1.3T  88% /data
/dev/bcache1    8.0T  6.6T  1.4T  83% /data1
/dev/bcache2    8.0T  7.0T  990G  88% /data2
*Node 11:*
/dev/bcache0     10T  8.1T  1.9T  82% /data
/dev/bcache1     10T  8.5T  1.5T  86% /data1
/dev/bcache2     10T  8.4T  1.6T  85% /data2
*Node 12:*
/dev/bcache0     10T  8.4T  1.6T  85% /data
/dev/bcache1     10T  8.4T  1.6T  85% /data1
/dev/bcache2     10T  8.2T  1.8T  83% /data2
*Node 13:*
/dev/bcache1     10T  8.7T  1.3T  88% /data1
/dev/bcache2     10T  8.8T  1.2T  88% /data2
/dev/bcache0     10T  8.6T  1.5T  86% /data




--
Regards,
Shreyansh Shah
AlphaGrep* Securities Pvt. Ltd.*

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux