Gluster : Improvements on "heal info" command

[Ashish Pandey <aspandey@xxxxxxxxxx>]

Hi All,

We have observed and heard from gluster users about the long time "heal info" command takes.

Even when we all want to know if a gluster volume is healthy or not, it takes time to list down all the files from all the bricks after which we can be

sure if the volume is healthy or not.

Here, we have come up with some options for "heal info" command which provide report quickly and reliably.

gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all]

--------

Problem: "gluster v heal <volname> info" command picks each subvolume and checks the .glusterfs/indices/xattrop folder of  every brick of that subvolume to find out if there is any entry
which needs to be healed. It picks the entry and takes a lock  on that entry to check xattrs to find out if that entry actually needs heal or not.

This LOCK->CHECK-XATTR->UNLOCK cycle takes lot of time for each file.

Let's consider two most often seen cases for which we use "heal info" and try to understand the improvements.

Case -1 : Consider 4+2 EC volume and all the bricks on 6 different nodes.

A brick of the volume is down and client has written 10000 files on one of the mount point of this volume. Entries for these 10K files will be created on ".glusterfs/indices/xattrop"

on all the rest of 5 bricks. Now, brick is UP and when we use "heal info" command for this volume, it goes to all the bricks and picks these 10K file entries and
goes through LOCK->CHECK-XATTR->UNLOCK cycle for all the files. This happens for all the bricks, that means, we check 50K files and perform the LOCK->CHECK-XATTR->UNLOCK cycle 50K times,

while only 10K entries were sufficient to check. It is a very time consuming operation. If IO"s are happening one some of the new files, we check these files also which will add the time.

Here, all we wanted to know if our volume has been healed and healthy.

Solution : Whenever a brick goes down and comes up and when we use "heal info" command, our *main intention* is to find out if the volume is *healthy* or *unhealthy*. A volume is unhealthy even if one
file is not healthy. So, we should scan bricks one by one and as soon as we find that one brick is having some entries which require to be healed, we can come out and list the files and say the volume is not
healthy. No need to scan rest of the bricks. That's where "--brick=[one,all]" option has been introduced.

"gluster v heal vol info --brick=[one,all]"
"one" - It will scan the brick sequentially and as soon as it will find any unhealthy entries, it will list it out and stop scanning other bricks.
"all" - It will act just like current behavior and provide all the files from all the bricks. If we do not provide this option, default (current) behavior will be applicable.

Case -2 : Consider 24 X (4+2) EC volume. Let's say one brick from *only one* of the sub volume has been replaced and a heal has been triggered.

To know if the volume is in healthy state, we go to each brick of *each and every sub volume* and check if there are any entries in ".glusterfs/indices/xattrop" folder which need heal or not.

If we know which sub volume participated in brick replacement, we just need to check health of that sub volume and not query/check other sub volumes.

If several clients are writing number of files on this volume, an entry for each of these files will be created in  .glusterfs/indices/xattrop and "heal info'

command will go through LOCK->CHECK-XATTR->UNLOCK cycle to find out if these entries need heal or not which takes lot of time.

In addition to this a client will also see performance drop as it will have to release and take lock again.

Solution: Provide an option to mention number of sub volume for which we want to check heal info.

"gluster v heal vol info --subvol=<no of subvolume>  "
Here, --subvol will be given number of the subvolume we want to check.
Example:
"gluster v heal vol info --subvol=1 "

===================================

Performance Data -

A quick performance test done on standalone system.

Type: Distributed-Disperse
Volume ID: ea40eb13-d42c-431c-9c89-0153e834e67e
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (4 + 2) = 12
Transport-type: tcp
Bricks:
Brick1: apandey:/home/apandey/bricks/gluster/vol-1
Brick2: apandey:/home/apandey/bricks/gluster/vol-2
Brick3: apandey:/home/apandey/bricks/gluster/vol-3
Brick4: apandey:/home/apandey/bricks/gluster/vol-4
Brick5: apandey:/home/apandey/bricks/gluster/vol-5
Brick6: apandey:/home/apandey/bricks/gluster/vol-6
Brick7: apandey:/home/apandey/bricks/gluster/new-1
Brick8: apandey:/home/apandey/bricks/gluster/new-2
Brick9: apandey:/home/apandey/bricks/gluster/new-3
Brick10: apandey:/home/apandey/bricks/gluster/new-4
Brick11: apandey:/home/apandey/bricks/gluster/new-5
Brick12: apandey:/home/apandey/bricks/gluster/new-6

Just disabled the shd to get the data -

Killed one brick each from two subvolumes and wrote 2000 files on mount point.

[root@apandey vol]# for i in {1..2000};do echo abc >> file-$i; done

Start the volume using force option and get the heal info. Following is the data -

[root@apandey glusterfs]# time gluster v heal vol info --brick=one >> /dev/null             <<<<<<<< This will scan brick one by one and come out as soon as we find volume is unhealthy.

real    0m8.316s
user    0m2.241s
sys    0m1.278s
[root@apandey glusterfs]#

[root@apandey glusterfs]# time gluster v heal vol info >> /dev/null                                  <<<<<<<< This is current behavior.

real    0m26.097s
user    0m10.868s
sys    0m6.198s
[root@apandey glusterfs]#
===================================

I would like your comments/suggestions on this improvements.

Specially, would like to hear on the new syntax of the command -

gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all]

Note that if we do not provide new options, command will behave just like it does right now.

Also, this improvement is valid for AFR and EC.

---

Ashish

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel