I found the issue.
The CentOS 7 RPMs, upon upgrade, modifies the .vol files. Among other
things, it adds "option shared-brick-count \d", using the number of
bricks in the volume.
This gives you an average free space per brick, instead of total free
space in the volume.
When I create a new volume, the value of "shared-brick-count" is "1".
find /var/lib/glusterd/vols -type f|xargs sed -i -e 's/option
shared-brick-count [0-9]*/option shared-brick-count 1/g'
On 09/27/17 17:09, Robert Hajime Lanning wrote:
Hi,
When I upgraded my cluster, df started returning some odd numbers for
my legacy volumes.
Newly created volumes after the upgrade, df works just fine.
I have been researching since Monday and have not found any reference
to this symptom.
"vm-images" is the old legacy volume, "test" is the new one.
[root@st-srv-03 ~]# (df -h|grep bricks;ssh st-srv-02 'df -h|grep
bricks')|sort
/dev/sda1 7.3T 991G 6.4T 14% /bricks/sda1
/dev/sda1 7.3T 991G 6.4T 14% /bricks/sda1
/dev/sdb1 7.3T 557G 6.8T 8% /bricks/sdb1
/dev/sdb1 7.3T 557G 6.8T 8% /bricks/sdb1
/dev/sdc1 7.3T 630G 6.7T 9% /bricks/sdc1
/dev/sdc1 7.3T 630G 6.7T 9% /bricks/sdc1
/dev/sdd1 7.3T 683G 6.7T 10% /bricks/sdd1
/dev/sdd1 7.3T 683G 6.7T 10% /bricks/sdd1
/dev/sde1 7.3T 657G 6.7T 9% /bricks/sde1
/dev/sde1 7.3T 658G 6.7T 9% /bricks/sde1
/dev/sdf1 7.3T 711G 6.6T 10% /bricks/sdf1
/dev/sdf1 7.3T 711G 6.6T 10% /bricks/sdf1
/dev/sdg1 7.3T 756G 6.6T 11% /bricks/sdg1
/dev/sdg1 7.3T 756G 6.6T 11% /bricks/sdg1
/dev/sdh1 7.3T 753G 6.6T 11% /bricks/sdh1
/dev/sdh1 7.3T 753G 6.6T 11% /bricks/sdh1
[root@st-srv-03 ~]# df -h|grep localhost
localhost:/test 59T 5.7T 53T 10% /gfs/test
localhost:/vm-images 7.3T 717G 6.6T 10% /gfs/vm-images
This is on CentOS 7.
Upgrade method was to shutdown glusterd/glusterfsd, "yum erase
centos-release-gluster310", "yum install centos-release-gluster312",
"yum upgrade -y", start glusterd.
[root@st-srv-03 ~]# rpm -qa|grep gluster
glusterfs-cli-3.12.1-1.el7.x86_64
glusterfs-3.12.1-1.el7.x86_64
nfs-ganesha-gluster-2.5.2-1.el7.x86_64
glusterfs-client-xlators-3.12.1-1.el7.x86_64
glusterfs-server-3.12.1-1.el7.x86_64
glusterfs-libs-3.12.1-1.el7.x86_64
glusterfs-api-3.12.1-1.el7.x86_64
glusterfs-fuse-3.12.1-1.el7.x86_64
centos-release-gluster312-1.0-1.el7.centos.noarch
[root@st-srv-03 ~]# gluster volume info test
Volume Name: test
Type: Distributed-Replicate
Volume ID: b53e0836-575e-46fd-9f86-ab7bf7c07ca9
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x 2 = 16
Transport-type: tcp
Bricks:
Brick1: st-srv-02-stor:/bricks/sda1/test
Brick2: st-srv-03-stor:/bricks/sda1/test
Brick3: st-srv-02-stor:/bricks/sdb1/test
Brick4: st-srv-03-stor:/bricks/sdb1/test
Brick5: st-srv-02-stor:/bricks/sdc1/test
Brick6: st-srv-03-stor:/bricks/sdc1/test
Brick7: st-srv-02-stor:/bricks/sdd1/test
Brick8: st-srv-03-stor:/bricks/sdd1/test
Brick9: st-srv-02-stor:/bricks/sde1/test
Brick10: st-srv-03-stor:/bricks/sde1/test
Brick11: st-srv-02-stor:/bricks/sdf1/test
Brick12: st-srv-03-stor:/bricks/sdf1/test
Brick13: st-srv-02-stor:/bricks/sdg1/test
Brick14: st-srv-03-stor:/bricks/sdg1/test
Brick15: st-srv-02-stor:/bricks/sdh1/test
Brick16: st-srv-03-stor:/bricks/sdh1/test
Options Reconfigured:
features.cache-invalidation: on
server.allow-insecure: on
auth.allow: 192.168.60.*
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
nfs-ganesha: disable
[root@st-srv-03 ~]# gluster volume info vm-images
Volume Name: vm-images
Type: Distributed-Replicate
Volume ID: 066a0598-e72e-419f-809e-86fa17f6f81c
Status: Started
Snapshot Count: 0
Number of Bricks: 8 x 2 = 16
Transport-type: tcp
Bricks:
Brick1: st-srv-02-stor:/bricks/sda1/vm-images
Brick2: st-srv-03-stor:/bricks/sda1/vm-images
Brick3: st-srv-02-stor:/bricks/sdb1/vm-images
Brick4: st-srv-03-stor:/bricks/sdb1/vm-images
Brick5: st-srv-02-stor:/bricks/sdc1/vm-images
Brick6: st-srv-03-stor:/bricks/sdc1/vm-images
Brick7: st-srv-02-stor:/bricks/sdd1/vm-images
Brick8: st-srv-03-stor:/bricks/sdd1/vm-images
Brick9: st-srv-02-stor:/bricks/sde1/vm-images
Brick10: st-srv-03-stor:/bricks/sde1/vm-images
Brick11: st-srv-02-stor:/bricks/sdf1/vm-images
Brick12: st-srv-03-stor:/bricks/sdf1/vm-images
Brick13: st-srv-02-stor:/bricks/sdg1/vm-images
Brick14: st-srv-03-stor:/bricks/sdg1/vm-images
Brick15: st-srv-02-stor:/bricks/sdh1/vm-images
Brick16: st-srv-03-stor:/bricks/sdh1/vm-images
Options Reconfigured:
features.cache-invalidation: on
server.allow-insecure: on
auth.allow: 192.168.60.*
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
nfs-ganesha: disable
--
Mr. Flibble
King of the Potato People
http://www.linkedin.com/in/RobertLanning
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
