Re: Speed up heal performance

[Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>]

On 10/14/2015 10:43 PM, Mohamed Pakkeer wrote:

Hi Pranith,

Will this patch improve the heal performance on distributed disperse volume?. Currently we are getting 10MB/s heal performance on 10G backed network. SHD daemon takes 5 days to complete the heal operation for single 4TB( 3.5 TB data) disk failure.

I will try to see if the patch for replication can be generalized for disperse volume as well. Keep watching the mailing list for updates :-)

Pranith

Regards,

Backer

On Wed, Oct 14, 2015 at 9:08 PM, Ben Turner <bturner@xxxxxxxxxx> wrote:

----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
> To: "Ben Turner" <bturner@xxxxxxxxxx>, "Humble Devassy Chirammal" <humble.devassy@xxxxxxxxx>, "Atin Mukherjee"
> <atin.mukherjee83@xxxxxxxxx>
> Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
> Sent: Wednesday, October 14, 2015 1:39:14 AM
> Subject: Re: Speed up heal performance
>
>
>
> On 10/13/2015 07:11 PM, Ben Turner wrote:
> > ----- Original Message -----
> >> From: "Humble Devassy Chirammal" <humble.devassy@xxxxxxxxx>
> >> To: "Atin Mukherjee" <atin.mukherjee83@xxxxxxxxx>
> >> Cc: "Ben Turner" <bturner@xxxxxxxxxx>, "gluster-users"
> >> <gluster-users@xxxxxxxxxxx>
> >> Sent: Tuesday, October 13, 2015 6:14:46 AM
> >> Subject: Re: Speed up heal performance
> >>
> >>> Good news is we already have a WIP patch review.glusterd.org/10851 to
> >> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for
> >> this. IIRC, we also have a BZ for the same
> >> Isnt it the same bugzilla (
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the
> >> commit log?
> > @Lindsay - No need for a BZ, the above BZ should suffice.
> >
> > @Anyone - In the commit I see:
> >
> >          { .key        = "cluster.shd-max-threads",
> >            .voltype    = "cluster/replicate",
> >            .option     = "shd-max-threads",
> >            .op_version = 1,
> >            .flags      = OPT_FLAG_CLIENT_OPT
> >          },
> >          { .key        = "cluster.shd-thread-batch-size",
> >            .voltype    = "cluster/replicate",
> >            .option     = "shd-thread-batch-size",
> >            .op_version = 1,
> >            .flags      = OPT_FLAG_CLIENT_OPT
> >          },
> >
> > So we can tune max threads and thread batch size?  I understand max
> > threads, but what is batch size?  In my testing on 10G NICs with a backend
> > that will service 10G throughput I see about 1.5 GB per minute of SH
> > throughput.  To Lindsay's other point, will this patch improve SH
> > throughput?  My systems can write at 1.5 GB / Sec and NICs can to 1.2 GB /
> > sec but I only see ~1.5 GB per _minute_ of SH throughput.  If we can not
> > only make SH multi threaded, but improve the performance of a single
> > thread that would be awesome.  Super bonus points if we can have some sort
> > of tunible that can limit the bandwidth each thread can consume.  It would
> > be great to be able to crank things up when the systems aren't busy and
> > slow things down when load increases.
> This patch is not merged because I thought we needed throttling feature
> to go in before we can merge this for better control of the self-heal
> speed. We are doing that for 3.8. So expect to see both of these for 3.8.

Great news!  You da man Pranith, next time I am on your side of the world beers are on me :)

-b

>
> Pranith
> >
> > -b
> >
> >
> >> --Humble
> >>
> >>
> >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee
> >> <atin.mukherjee83@xxxxxxxxx>
> >> wrote:
> >>
> >>> -Atin
> >>> Sent from one plus one
> >>> On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner@xxxxxxxxxx> wrote:
> >>>> ----- Original Message -----
> >>>>> From: "Lindsay Mathieson" <lindsay.mathieson@xxxxxxxxx>
> >>>>> To: "gluster-users" <gluster-users@xxxxxxxxxxx>
> >>>>> Sent: Friday, October 9, 2015 9:18:11 AM
> >>>>> Subject: Speed up heal performance
> >>>>>
> >>>>> Is there any way to max out heal performance? My cluster is unused
> >>> overnight,
> >>>>> and lightly used at lunchtimes, it would be handy to speed up a heal.
> >>>>>
> >>>>> The only tuneable I found was cluster.self-heal-window-size, which
> >>> doesn't
> >>>>> seem to make much difference.
> >>>> I don't know of any way to speed this up, maybe someone else could chime
> >>> in here that knows the heal daemon better than me.  Maybe you could open
> >>> an
> >>> RFE on this?  In my testing I only see 2 files getting healed at a time
> >>> per
> >>> replica pair.  I would like to see this be multi threaded(if its not
> >>> already) with the ability to tune it to control resource usage(similar to
> >>> what we did in the rebalance refactoring done recently).  If you let me
> >>> know the BZ # I'll add my data + suggestions, I have been testing this
> >>> pretty extensively in recent weeks and good data + some ideas on how to
> >>> speed things up.
> >>> Good news is we already have a WIP patch review.glusterd.org/10851 to
> >>> introduce multi threaded shd. Credits to Richard/Shreyas from facebook
> >>> for
> >>> this. IIRC, we also have a BZ for the same but the patch is in rfc as of
> >>> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct
> >>> me otherwise.
> >>>> -b
> >>>>
> >>>>> thanks,
> >>>>> --
> >>>>> Lindsay
> >>>>>
> >>>>> _______________________________________________
> >>>>> Gluster-users mailing list
> >>>>> Gluster-users@xxxxxxxxxxx
> >>>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>> _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users@xxxxxxxxxxx
> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users@xxxxxxxxxxx
> >>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users@xxxxxxxxxxx
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users