Re: df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Jose,

There is a known issue with gluster 3.12.x builds (see [1]) so you may be running into this.

The "shared-brick-count" values seem fine on stor1. Please send us "grep -n "share" /var/lib/glusterd/vols/volumedisk1/*" results for the other nodes so we can check if they are the cause.


Regards,
Nithya



[1] https://bugzilla.redhat.com/show_bug.cgi?id=1517260

On 28 February 2018 at 03:03, Jose V. Carrión <jocarbur@xxxxxxxxx> wrote:

Hi,

Some days ago all my glusterfs configuration was working fine. Today I realized that the total size reported by df command was changed and is smaller than the aggregated capacity of all the bricks in the volume.

I checked that all the volumes status are fine, all the glusterd daemons are running, there is no error in logs,  however df shows a bad total size.

My configuration for one volume: volumedisk1

[root@stor1 ~]# gluster volume status volumedisk1  detail

Status of volume: volumedisk1
------------------------------------------------------------------------------
Brick                : Brick stor1data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 13579              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.0TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273123069         
------------------------------------------------------------------------------
Brick                : Brick stor2data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 13344              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.0TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273124718         
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
TCP Port             : 49154              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 17439              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.7TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273125437         
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
TCP Port             : 49155              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 17459              
File System          : xfs                
Device               : /dev/sdd1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.6TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273127036         
 

Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB = 196,4 TB  but df shows:

[root@stor1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              48G   21G   25G  46% /
tmpfs                  32G   80K   32G   1% /dev/shm
/dev/sda1             190M   62M  119M  35% /boot
/dev/sda4             395G  251G  124G  68% /data
/dev/sdb1              26T  601G   25T   3% /mnt/glusterfs/vol0
/dev/sdc1              50T   15T   36T  29% /mnt/glusterfs/vol1
stor1data:/volumedisk0
                       76T  1,6T   74T   3% /volumedisk0
stor1data:/volumedisk1
                      148T   42T  106T  29% /volumedisk1

Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB

It's a production system so I hope you can help me.

Thanks in advance.

Jose V.


Below some other data of my configuration:

[root@stor1 ~]# gluster volume info
 
Volume Name: volumedisk0
Type: Distribute
Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol0/brick1
Brick2: stor2data:/mnt/glusterfs/vol0/brick1
Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
Options Reconfigured:
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on
 
Volume Name: volumedisk1
Type: Distribute
Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol1/brick1
Brick2: stor2data:/mnt/glusterfs/vol1/brick1
Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux