Re: df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi Jose,

On 28 February 2018 at 18:28, Jose V. Carrión <jocarbur@xxxxxxxxx> wrote:
Hi Nithya,

I applied the workarround for this bug and now df shows the right size:

That is good to hear.

 
[root@stor1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1              50T   16T   34T  33% /mnt/glusterfs/vol1
stor1data:/volumedisk0
                      101T  3,3T   97T   4% /volumedisk0
stor1data:/volumedisk1
                      197T   61T  136T  31% /volumedisk1


[root@stor2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1              50T   16T   34T  33% /mnt/glusterfs/vol1
stor2data:/volumedisk0
                      101T  3,3T   97T   4% /volumedisk0
stor2data:/volumedisk1
                      197T   61T  136T  31% /volumedisk1


[root@stor3 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1              25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
/dev/sdb2              25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
/dev/sdc1              50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
/dev/sdd1              50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1
stor3data:/volumedisk0
                      101T  3,3T   97T   4% /volumedisk0
stor3data:/volumedisk1
                      197T   61T  136T  31% /volumedisk1


However I'm concerned because, as you can see, the volumedisk0 on stor3data is composed by 2 bricks on thesame disk but on different partitions (/dev/sdb1 and /dev/sdb2).
After to aplly the workarround, the  shared-brick-count parameter was setted to 1 in all the bricks and all the servers (see below). Could be this an issue ?

No, this is correct. The shared-brick-count will be > 1 only if multiple bricks share the same partition.

 
Also, I can check that stor3data is now unbalanced respect stor1data and stor2data. The three nodes have the same size of brick but stor3data bricks have used 1TB less than stor1data and stor2data:


This does not necessarily indicate a problem. The distribution need not be exactly equal and depends on the filenames. Can you provide more information on the kind of dataset (how many files, sizes etc) on this volume? Did you create the volume with all 4 bricks or add some later?

Regards,
Nithya

stor1data:
/dev/sdb1              26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1              50T   16T   34T  33% /mnt/glusterfs/vol1

stor2data bricks:
/dev/sdb1              26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1              50T   16T   34T  33% /mnt/glusterfs/vol1

stor3data bricks:
  /dev/sdb1              25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
  /dev/sdb2              25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
   dev/sdc1              50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
   /dev/sdd1             50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1


[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0

[root@stor2 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0

[root@stor3t ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0

Thaks for your help,
Greetings.

Jose V.


2018-02-28 5:07 GMT+01:00 Nithya Balachandran <nbalacha@xxxxxxxxxx>:
Hi Jose,

There is a known issue with gluster 3.12.x builds (see [1]) so you may be running into this.

The "shared-brick-count" values seem fine on stor1. Please send us "grep -n "share" /var/lib/glusterd/vols/volumedisk1/*" results for the other nodes so we can check if they are the cause.


Regards,
Nithya




On 28 February 2018 at 03:03, Jose V. Carrión <jocarbur@xxxxxxxxx> wrote:

Hi,

Some days ago all my glusterfs configuration was working fine. Today I realized that the total size reported by df command was changed and is smaller than the aggregated capacity of all the bricks in the volume.

I checked that all the volumes status are fine, all the glusterd daemons are running, there is no error in logs,  however df shows a bad total size.

My configuration for one volume: volumedisk1

[root@stor1 ~]# gluster volume status volumedisk1  detail

Status of volume: volumedisk1
------------------------------------------------------------------------------
Brick                : Brick stor1data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 13579              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.0TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273123069         
------------------------------------------------------------------------------
Brick                : Brick stor2data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 13344              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.0TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273124718         
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
TCP Port             : 49154              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 17439              
File System          : xfs                
Device               : /dev/sdc1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.7TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273125437         
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
TCP Port             : 49155              
RDMA Port            : 0                  
Online               : Y                  
Pid                  : 17459              
File System          : xfs                
Device               : /dev/sdd1          
Mount Options        : rw,noatime         
Inode Size           : 512                
Disk Space Free      : 35.6TB             
Total Disk Space     : 49.1TB             
Inode Count          : 5273970048         
Free Inodes          : 5273127036         
 

Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB = 196,4 TB  but df shows:

[root@stor1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              48G   21G   25G  46% /
tmpfs                  32G   80K   32G   1% /dev/shm
/dev/sda1             190M   62M  119M  35% /boot
/dev/sda4             395G  251G  124G  68% /data
/dev/sdb1              26T  601G   25T   3% /mnt/glusterfs/vol0
/dev/sdc1              50T   15T   36T  29% /mnt/glusterfs/vol1
stor1data:/volumedisk0
                       76T  1,6T   74T   3% /volumedisk0
stor1data:/volumedisk1
                      148T   42T  106T  29% /volumedisk1

Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB

It's a production system so I hope you can help me.

Thanks in advance.

Jose V.


Below some other data of my configuration:

[root@stor1 ~]# gluster volume info
 
Volume Name: volumedisk0
Type: Distribute
Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol0/brick1
Brick2: stor2data:/mnt/glusterfs/vol0/brick1
Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
Options Reconfigured:
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on
 
Volume Name: volumedisk1
Type: Distribute
Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol1/brick1
Brick2: stor2data:/mnt/glusterfs/vol1/brick1
Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:    option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:    option shared-brick-count 0



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users



_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux