Hi,
Some days ago all my glusterfs configuration was working fine. Today I realized that the total size reported by df command was changed and is smaller than the aggregated capacity of all the bricks in the volume.
I checked that all the volumes status are fine, all the glusterd
daemons are running, there is no error in logs, however df shows a bad
total size.
My configuration for one volume: volumedisk1
[root@stor1 ~]# gluster volume status volumedisk1 detailStatus of volume: volumedisk1
------------------------------------------------------------------------------
Brick : Brick stor1data:/mnt/glusterfs/vol1/brick1
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 13579
File System : xfs
Device : /dev/sdc1
Mount Options : rw,noatime
Inode Size : 512
Disk Space Free : 35.0TB
Total Disk Space : 49.1TB
Inode Count : 5273970048
Free Inodes : 5273123069
------------------------------------------------------------------------------
Brick : Brick stor2data:/mnt/glusterfs/vol1/brick1
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 13344
File System : xfs
Device : /dev/sdc1
Mount Options : rw,noatime
Inode Size : 512
Disk Space Free : 35.0TB
Total Disk Space : 49.1TB
Inode Count : 5273970048
Free Inodes : 5273124718
------------------------------------------------------------------------------
Brick : Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17439
File System : xfs
Device : /dev/sdc1
Mount Options : rw,noatime
Inode Size : 512
Disk Space Free : 35.7TB
Total Disk Space : 49.1TB
Inode Count : 5273970048
Free Inodes : 5273125437
------------------------------------------------------------------------------
Brick : Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17459
File System : xfs
Device : /dev/sdd1
Mount Options : rw,noatime
Inode Size : 512
Disk Space Free : 35.6TB
Total Disk Space : 49.1TB
Inode Count : 5273970048
Free Inodes : 5273127036
Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB = 196,4 TB but df shows:
[root@stor1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 48G 21G 25G 46% /
tmpfs 32G 80K 32G 1% /dev/shm
/dev/sda1 190M 62M 119M 35% /boot
/dev/sda4 395G 251G 124G 68% /data
/dev/sdb1 26T 601G 25T 3% /mnt/glusterfs/vol0
/dev/sdc1 50T 15T 36T 29% /mnt/glusterfs/vol1
stor1data:/volumedisk0
76T 1,6T 74T 3% /volumedisk0
stor1data:/volumedisk1
148T 42T 106T 29% /volumedisk1
Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB
It's a production system so I hope you can help me.
Thanks in advance.
Jose V.
Below some other data of my configuration:
[root@stor1 ~]# gluster volume info
Volume Name: volumedisk0
Type: Distribute
Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol0/brick1
Brick2: stor2data:/mnt/glusterfs/vol0/brick1
Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
Options Reconfigured:
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on
Volume Name: volumedisk1
Type: Distribute
Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol1/brick1
Brick2: stor2data:/mnt/glusterfs/vol1/brick1
Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on
[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3: option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3: option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3: option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3: option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3: option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3: option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 0
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
