Gluster 9.3 not distributing data evenly across bricks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,
We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We are noticing uneven data distribution across the bricks. Some bricks are 100% utilised whereas some are < 85%. Due to this, data write randomly fails with error OSError(28, 'No space left on device') even when the volume has sufficient space on other bricks. We expect the glusterfs process to automatically sync and maintain equal data distribution across the bricks.

This is the volume info:
root@nas-5:~# gluster v info
 
Volume Name: datavol
Type: Distribute
Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
Status: Started
Snapshot Count: 0
Number of Bricks: 47
Transport-type: tcp
Bricks:
Brick1: 10.132.2.101:/data/data
Brick2: 10.132.2.101:/data1/data
Brick3: 10.132.2.101:/data2/data
Brick4: 10.132.2.101:/data3/data
Brick5: 10.132.2.102:/data/data
Brick6: 10.132.2.102:/data1/data
Brick7: 10.132.2.102:/data2/data
Brick8: 10.132.2.102:/data3/data
Brick9: 10.132.2.103:/data/data
Brick10: 10.132.2.103:/data1/data
Brick11: 10.132.2.103:/data2/data
Brick12: 10.132.2.103:/data3/data
Brick13: 10.132.2.104:/data/data
Brick14: 10.132.2.104:/data3/data
Brick15: 10.132.2.105:/data1/data
Brick16: 10.132.2.105:/data2/data
Brick17: 10.132.2.106:/data/data
Brick18: 10.132.2.106:/data1/data
Brick19: 10.132.2.106:/data2/data
Brick20: 10.132.2.107:/data/data
Brick21: 10.132.2.107:/data1/data
Brick22: 10.132.2.107:/data2/data
Brick23: 10.132.2.108:/data/data
Brick24: 10.132.2.108:/data1/data
Brick25: 10.132.2.108:/data2/data
Brick26: 10.132.2.109:/data/data
Brick27: 10.132.2.109:/data1/data
Brick28: 10.132.2.109:/data2/data
Brick29: 10.132.2.110:/data/data
Brick30: 10.132.2.110:/data1/data
Brick31: 10.132.2.111:/data/data
Brick32: 10.132.2.111:/data1/data
Brick33: 10.132.2.111:/data2/data
Brick34: 10.132.2.112:/data/data
Brick35: 10.132.2.112:/data1/data
Brick36: 10.132.2.112:/data2/data
Brick37: 10.132.2.113:/data/data
Brick38: 10.132.2.113:/data1/data
Brick39: 10.132.2.113:/data2/data
Brick40: 10.132.2.108:/data3/data
Brick41: 10.132.2.107:/data3/data
Brick42: 10.132.2.106:/data3/data
Brick43: 10.132.2.105:/data3/data
Brick44: 10.132.2.110:/data2/data
Brick45: 10.132.2.105:/data/data
Brick46: 10.132.2.104:/data1/data
Brick47: 10.132.2.104:/data2/data
Options Reconfigured:
performance.client-io-threads: on
cluster.min-free-disk: 2%
performance.cache-refresh-timeout: 60
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 90
storage.health-check-interval: 60
storage.health-check-timeout: 60
performance.io-cache-size: 8GB
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
root@nas-5:~# glusterfs --version
glusterfs 9.3
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
root@nas-5:~#


This is the disk usage distribution when the write failed:
Node 1:
/dev/bcache1    9.7T  8.5T  1.2T  88% /data
/dev/bcache3    3.9T  3.3T  563G  86% /data1
/dev/bcache0    3.9T  3.3T  560G  86% /data2
/dev/bcache2    3.9T  3.5T  395G  90% /data3
Node 2:
/dev/bcache0    9.7T  8.5T  1.2T  88% /data
/dev/bcache1    3.9T  3.5T  421G  90% /data1
/dev/bcache2    3.9T  3.6T  330G  92% /data2
/dev/bcache3    3.9T  3.4T  513G  87% /data3
Node 3:
/dev/bcache0    9.7T  8.8T  882G  92% /data
/dev/bcache1    3.9T  3.6T  335G  92% /data1
/dev/bcache2    3.9T  3.4T  532G  87% /data2
/dev/bcache3    3.9T  3.3T  564G  86% /data3
Node 4:
/dev/bcache0    9.7T  8.7T  982G  91% /data
/dev/bcache1    3.9T  3.5T  424G  90% /data1
/dev/bcache2    3.9T  3.4T  549G  87% /data2
/dev/bcache3    3.9T  3.6T  344G  92% /data3
Node 5:
/dev/bcache0    9.7T  8.5T  1.3T  88% /data
/dev/bcache1    3.9T  3.6T  288G  93% /data1
/dev/bcache2    3.9T  3.4T  470G  89% /data2
/dev/bcache3    9.9T  9.8T  101G 100% /data3
Node 6:
/dev/bcache0    9.7T  8.2T  1.5T  86% /data
/dev/bcache1    3.9T  3.4T  526G  87% /data1
/dev/bcache2    3.9T  3.5T  431G  90% /data2
/dev/bcache3    9.9T  8.9T  1.1T  90% /data3
Node 7:
/dev/bcache0    9.7T  8.9T  783G  93% /data
/dev/bcache1    3.9T  3.3T  561G  86% /data1
/dev/bcache2    3.9T  3.5T  360G  91% /data2
/dev/bcache3    9.9T  8.7T  1.2T  89% /data3
Node 8:
/dev/bcache0    9.7T  8.7T  994G  90% /data
/dev/bcache1    3.9T  3.3T  645G  84% /data1
/dev/bcache2    3.9T  3.4T  519G  87% /data2
/dev/bcache3    9.9T  9.0T  868G  92% /data3
Node 9:
/dev/bcache0     10T  8.6T  1.4T  87% /data
/dev/bcache1    8.0T  6.7T  1.4T  84% /data1
/dev/bcache2    8.0T  6.8T  1.3T  85% /data2
Node 10:
/dev/bcache0     10T  8.8T  1.3T  88% /data
/dev/bcache1    8.0T  6.6T  1.4T  83% /data1
/dev/bcache2    8.0T  7.0T  990G  88% /data2
Node 11:
/dev/bcache0     10T  8.1T  1.9T  82% /data
/dev/bcache1     10T  8.5T  1.5T  86% /data1
/dev/bcache2     10T  8.4T  1.6T  85% /data2
Node 12:
/dev/bcache0     10T  8.4T  1.6T  85% /data
/dev/bcache1     10T  8.4T  1.6T  85% /data1
/dev/bcache2     10T  8.2T  1.8T  83% /data2
Node 13:
/dev/bcache1     10T  8.7T  1.3T  88% /data1
/dev/bcache2     10T  8.8T  1.2T  88% /data2
/dev/bcache0     10T  8.6T  1.5T  86% /data




--
Regards,
Shreyansh Shah

AlphaGrep Securities Pvt. Ltd.
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux