Hi all, I hope this message finds you well.
I've been running a Gluster volume (32 bricks in distributed, replicated mode) on my 2 home servers for about 1,5 years now. I'm generally very happy with it!
Disks are distributed across 4 enclosures (2 enclosures per server). At one point one of these enclosures failed (8 bricks down on 1 server) but due to the awesomeness of Gluster (and my lack of monitoring :-( ) I only noticed this after about 6 weeks... This left me with A LOT of pending heals, about 40k per brick if I remember correctly.
Well, I brought back online the failed bricks and let the the gluster heal. And it did, mostly... It left about 1 to 4 pending heals on multiple bricks that won't heal, no matter what I've tried. I just let them be for some time until I had time to figure out what to do with them.
Also, because some disks were about to fail, I started replacing bricks. And taking advantage of this, I replaced them with a larger disk (4TB -> 8TB). Healing took care of copying all data to the new brick and finished succesfully. However, for some reason I do not see an increase in total space on the systems were I have mounted the Gluster volume.
So in short, I have 2 issues:
- Some Pending Heals that I cannot get rid of
- Gluster total space being reported incorrect
Ideally, I'd like to address issue 2 first (seems the easiest fix), and then focus on issue 1. Should that be ok?
I hope you guys can help me with these 2 issues. Thanks in advance!
Added as attachent:
- Commands used to replace the brick
- "gluster volume status data detail" output
- "grep -n "shared-brick-count" /var/lib/glusterd/vols/data/*" output, as I read somewhere that this may be relevant...
--
groet / cheers,
Patrick Dijkgraaf
[root@dorus ~]# gluster volume status data detail Status of volume: data ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/01/VGK8VEXG TCP Port : 50816 RDMA Port : 0 Online : Y Pid : 134061 File System : xfs Device : /dev/sda2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781049061 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/01/VRK4L7MK TCP Port : 54462 RDMA Port : 0 Online : Y Pid : 146792 File System : xfs Device : /dev/sda2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781049061 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/02/VKKP6LAY TCP Port : 53275 RDMA Port : 0 Online : Y Pid : 134066 File System : xfs Device : /dev/sdb2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984850 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/02/VAG4UNTL TCP Port : 52386 RDMA Port : 0 Online : Y Pid : 146797 File System : xfs Device : /dev/sdb2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984847 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/03/WD-WCC7K7PCN1FH TCP Port : 56176 RDMA Port : 0 Online : Y Pid : 134090 File System : xfs Device : /dev/sdc2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 793.4GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345631 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/03/WD-WCC7K2ZF967P TCP Port : 51349 RDMA Port : 0 Online : Y Pid : 146821 File System : xfs Device : /dev/sdc2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 793.4GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345632 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/04/WD-WCC4E6XXFZK5 TCP Port : 59307 RDMA Port : 0 Online : Y Pid : 134114 File System : xfs Device : /dev/sdd2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 700.0GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390344717 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/04/WD-WCC4E4DEL2R7 TCP Port : 50401 RDMA Port : 0 Online : Y Pid : 146845 File System : xfs Device : /dev/sdd2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 700.0GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390344716 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/05/VK0RBEUY TCP Port : 52860 RDMA Port : 0 Online : Y Pid : 134137 File System : xfs Device : /dev/sde2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780986042 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/05/VKKRGDRY TCP Port : 49846 RDMA Port : 0 Online : Y Pid : 146869 File System : xfs Device : /dev/sde2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780986043 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/06/R6GLYPZY TCP Port : 57091 RDMA Port : 0 Online : Y Pid : 134160 File System : xfs Device : /dev/sdf2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780985689 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/06/R6GUJRGY TCP Port : 53391 RDMA Port : 0 Online : Y Pid : 146891 File System : xfs Device : /dev/sdf2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780985687 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/07/WD-WCC7K3KZAAY7 TCP Port : 60264 RDMA Port : 0 Online : Y Pid : 134184 File System : xfs Device : /dev/sdg2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 738.4GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345263 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/07/WD-WCC4E2FRNVLS TCP Port : 56653 RDMA Port : 0 Online : Y Pid : 146933 File System : xfs Device : /dev/sdg2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 738.4GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345267 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/08/VKKNNE7Y TCP Port : 49829 RDMA Port : 0 Online : Y Pid : 134231 File System : xfs Device : /dev/sdh2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984701 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/08/VK0PTDWY TCP Port : 56208 RDMA Port : 0 Online : Y Pid : 146962 File System : xfs Device : /dev/sdh2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984676 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/09/CA0ZRMEK TCP Port : 52365 RDMA Port : 0 Online : Y Pid : 134259 File System : xfs Device : /dev/sdj2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781049832 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/09/VRK4L9PK TCP Port : 56532 RDMA Port : 0 Online : Y Pid : 146992 File System : xfs Device : /dev/sdj2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.1TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781049832 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/10/WD-WCC7K2ZF95D4 TCP Port : 60470 RDMA Port : 0 Online : Y Pid : 134265 File System : xfs Device : /dev/sdi2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 772.2GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390344692 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/10/CA0ZYD5K TCP Port : 60153 RDMA Port : 0 Online : Y Pid : 147482 File System : xfs Device : /dev/sdi2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 6.3TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781224223 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/11/VK0N9YMY TCP Port : 54556 RDMA Port : 0 Online : Y Pid : 134313 File System : xfs Device : /dev/sdk2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780985141 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/11/R6GSPRKY TCP Port : 60423 RDMA Port : 0 Online : Y Pid : 146997 File System : xfs Device : /dev/sdk2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.4TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780985144 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/12/WD-WCC4E4UREPNH TCP Port : 59959 RDMA Port : 0 Online : Y Pid : 134336 File System : xfs Device : /dev/sdl2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 505.3GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390344901 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/12/WD-WCC7K7PCNUDT TCP Port : 51963 RDMA Port : 0 Online : Y Pid : 147025 File System : xfs Device : /dev/sdl2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 505.3GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390344897 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/13/R6GV252Y TCP Port : 50713 RDMA Port : 0 Online : Y Pid : 134364 File System : xfs Device : /dev/sdm2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984671 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/13/VKKMP42Y TCP Port : 49773 RDMA Port : 0 Online : Y Pid : 147067 File System : xfs Device : /dev/sdm2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.3TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984814 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/14/VGJYYJXG TCP Port : 58434 RDMA Port : 0 Online : Y Pid : 134388 File System : xfs Device : /dev/sdn2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.0TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781046270 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/14/CA0VWVTK TCP Port : 54407 RDMA Port : 0 Online : Y Pid : 147075 File System : xfs Device : /dev/sdn2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 4.0TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 781050455 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/15/WD-WCC7K1HU54PX TCP Port : 60452 RDMA Port : 0 Online : Y Pid : 134415 File System : xfs Device : /dev/sdo2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 579.2GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345128 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/15/WD-WCC4E0UP9SXY TCP Port : 50382 RDMA Port : 0 Online : Y Pid : 147097 File System : xfs Device : /dev/sdo2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 599.8GB Total Disk Space : 3.6TB Inode Count : 390701440 Free Inodes : 390345230 ------------------------------------------------------------------------------ Brick : Brick dorus:/mnt/bricks/data/16/R6GV83HY TCP Port : 50703 RDMA Port : 0 Online : Y Pid : 134439 File System : xfs Device : /dev/sdp2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984648 ------------------------------------------------------------------------------ Brick : Brick cornelis:/mnt/bricks/data/16/VK0N9UBY TCP Port : 56717 RDMA Port : 0 Online : Y Pid : 147142 File System : xfs Device : /dev/sdp2 Mount Options : rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota Inode Size : 512 Disk Space Free : 1.2TB Total Disk Space : 7.3TB Inode Count : 781402176 Free Inodes : 780984644
[root@dorus ~]# grep -n "shared-brick-count" /var/lib/glusterd/vols/data/* grep: /var/lib/glusterd/vols/data/bricks: Is a directory /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-01-VRK4L7MK.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-02-VAG4UNTL.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-03-WD-WCC7K2ZF967P.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-04-WD-WCC4E4DEL2R7.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-05-VKKRGDRY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-06-R6GUJRGY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-07-WD-WCC4E2FRNVLS.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-08-VK0PTDWY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-09-VRK4L9PK.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-10-CA0ZYD5K.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-11-R6GSPRKY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-12-WD-WCC7K7PCNUDT.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-13-VKKMP42Y.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-14-CA0VWVTK.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-15-WD-WCC4E0UP9SXY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.cornelis.mnt-bricks-data-16-VK0N9UBY.vol:3: option shared-brick-count 0 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-01-VGK8VEXG.vol:3: option shared-brick-count 2 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-02-VKKP6LAY.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-03-WD-WCC7K7PCN1FH.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-04-WD-WCC4E6XXFZK5.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-05-VK0RBEUY.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-06-R6GLYPZY.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-07-WD-WCC7K3KZAAY7.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-08-VKKNNE7Y.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-09-CA0ZRMEK.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-10-WD-WCC7K2ZF95D4.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-11-VK0N9YMY.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-12-WD-WCC4E4UREPNH.vol:3: option shared-brick-count 2 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-13-R6GV252Y.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-14-VGJYYJXG.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-15-WD-WCC7K1HU54PX.vol:3: option shared-brick-count 1 /var/lib/glusterd/vols/data/data.dorus.mnt-bricks-data-16-R6GV83HY.vol:3: option shared-brick-count 1
VOLUME=data
DISKNR=10
NODE=${HOSTNAME}
NODELETTER=${NODE:0:1}
OLDSERIAL=`basename \`/usr/bin/gluster volume status all detail |grep -P "^(?=.*${NODE})(?=.*${VOLUME})(?=.*${DISKNR})" | awk '{print $4}'\``
## Unmount old disk
kill -15 `ps aux | grep glusterfsd |grep ${VOLUME} |grep ${DISKNR} |awk '{print $2}'`
sleep 10
umount /mnt/bricks/${VOLUME}/${DISKNR}
## Check that the gluster volume is online with 1 brick offline
gluster volume status ${VOLUME}
<replace disk>
NEWDISK=sdi
NEWSERIAL=`smartctl -a /dev/${NEWDISK} |grep Serial | awk '{print $3}' | sed -e 's/WD-//g'`
echo ${NEWSERIAL}
## Create partitions on the new disk
parted -a optimal /dev/${NEWDISK} 'mklabel gpt' 'mkpart primary 1MiB 3MiB' 'name 1 grub' 'set 1 bios_grub on' 'mkpart primary 3MiB -1' "name 2 ${NODELETTER}-${VOLUME}-${DISKNR}"
## Create File System on the new disks
mkfs.xfs -L ${NODELETTER}-${VOLUME}-${DISKNR} /dev/${NEWDISK}2
## Mount new disk
mount -a
## Check that all 16 disks are mounted
mount |grep data |wc -l
## Create the new brick base directory
mkdir -p /mnt/bricks/${VOLUME}/${DISKNR}/${NEWSERIAL}
## Replace the faulty brick
gluster volume replace-brick ${VOLUME} ${NODE}:/mnt/bricks/${VOLUME}/${DISKNR}/${OLDSERIAL} ${NODE}:/mnt/bricks/${VOLUME}/${DISKNR}/${NEWSERIAL} commit force
## Check that the new brick is now online
gluster volume status ${VOLUME}
## Check the pending heals and see then decrease over time
gluster volume heal ${VOLUME} info
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
