----- Original Message ----- > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > To: "Ben Turner" <bturner@xxxxxxxxxx>, "Humble Devassy Chirammal" <humble.devassy@xxxxxxxxx>, "Atin Mukherjee" > <atin.mukherjee83@xxxxxxxxx> > Cc: "gluster-users" <gluster-users@xxxxxxxxxxx> > Sent: Wednesday, October 14, 2015 1:39:14 AM > Subject: Re: Speed up heal performance > > > > On 10/13/2015 07:11 PM, Ben Turner wrote: > > ----- Original Message ----- > >> From: "Humble Devassy Chirammal" <humble.devassy@xxxxxxxxx> > >> To: "Atin Mukherjee" <atin.mukherjee83@xxxxxxxxx> > >> Cc: "Ben Turner" <bturner@xxxxxxxxxx>, "gluster-users" > >> <gluster-users@xxxxxxxxxxx> > >> Sent: Tuesday, October 13, 2015 6:14:46 AM > >> Subject: Re: Speed up heal performance > >> > >>> Good news is we already have a WIP patch review.glusterd.org/10851 to > >> introduce multi threaded shd. Credits to Richard/Shreyas from facebook for > >> this. IIRC, we also have a BZ for the same > >> Isnt it the same bugzilla ( > >> https://bugzilla.redhat.com/show_bug.cgi?id=1221737) mentioned in the > >> commit log? > > @Lindsay - No need for a BZ, the above BZ should suffice. > > > > @Anyone - In the commit I see: > > > > { .key = "cluster.shd-max-threads", > > .voltype = "cluster/replicate", > > .option = "shd-max-threads", > > .op_version = 1, > > .flags = OPT_FLAG_CLIENT_OPT > > }, > > { .key = "cluster.shd-thread-batch-size", > > .voltype = "cluster/replicate", > > .option = "shd-thread-batch-size", > > .op_version = 1, > > .flags = OPT_FLAG_CLIENT_OPT > > }, > > > > So we can tune max threads and thread batch size? I understand max > > threads, but what is batch size? In my testing on 10G NICs with a backend > > that will service 10G throughput I see about 1.5 GB per minute of SH > > throughput. To Lindsay's other point, will this patch improve SH > > throughput? My systems can write at 1.5 GB / Sec and NICs can to 1.2 GB / > > sec but I only see ~1.5 GB per _minute_ of SH throughput. If we can not > > only make SH multi threaded, but improve the performance of a single > > thread that would be awesome. Super bonus points if we can have some sort > > of tunible that can limit the bandwidth each thread can consume. It would > > be great to be able to crank things up when the systems aren't busy and > > slow things down when load increases. > This patch is not merged because I thought we needed throttling feature > to go in before we can merge this for better control of the self-heal > speed. We are doing that for 3.8. So expect to see both of these for 3.8. Great news! You da man Pranith, next time I am on your side of the world beers are on me :) -b > > Pranith > > > > -b > > > > > >> --Humble > >> > >> > >> On Tue, Oct 13, 2015 at 7:26 AM, Atin Mukherjee > >> <atin.mukherjee83@xxxxxxxxx> > >> wrote: > >> > >>> -Atin > >>> Sent from one plus one > >>> On Oct 13, 2015 3:16 AM, "Ben Turner" <bturner@xxxxxxxxxx> wrote: > >>>> ----- Original Message ----- > >>>>> From: "Lindsay Mathieson" <lindsay.mathieson@xxxxxxxxx> > >>>>> To: "gluster-users" <gluster-users@xxxxxxxxxxx> > >>>>> Sent: Friday, October 9, 2015 9:18:11 AM > >>>>> Subject: Speed up heal performance > >>>>> > >>>>> Is there any way to max out heal performance? My cluster is unused > >>> overnight, > >>>>> and lightly used at lunchtimes, it would be handy to speed up a heal. > >>>>> > >>>>> The only tuneable I found was cluster.self-heal-window-size, which > >>> doesn't > >>>>> seem to make much difference. > >>>> I don't know of any way to speed this up, maybe someone else could chime > >>> in here that knows the heal daemon better than me. Maybe you could open > >>> an > >>> RFE on this? In my testing I only see 2 files getting healed at a time > >>> per > >>> replica pair. I would like to see this be multi threaded(if its not > >>> already) with the ability to tune it to control resource usage(similar to > >>> what we did in the rebalance refactoring done recently). If you let me > >>> know the BZ # I'll add my data + suggestions, I have been testing this > >>> pretty extensively in recent weeks and good data + some ideas on how to > >>> speed things up. > >>> Good news is we already have a WIP patch review.glusterd.org/10851 to > >>> introduce multi threaded shd. Credits to Richard/Shreyas from facebook > >>> for > >>> this. IIRC, we also have a BZ for the same but the patch is in rfc as of > >>> now. AFAIK, this is a candidate to land in 3.8 as well, Vijay can correct > >>> me otherwise. > >>>> -b > >>>> > >>>>> thanks, > >>>>> -- > >>>>> Lindsay > >>>>> > >>>>> _______________________________________________ > >>>>> Gluster-users mailing list > >>>>> Gluster-users@xxxxxxxxxxx > >>>>> http://www.gluster.org/mailman/listinfo/gluster-users > >>>> _______________________________________________ > >>>> Gluster-users mailing list > >>>> Gluster-users@xxxxxxxxxxx > >>>> http://www.gluster.org/mailman/listinfo/gluster-users > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users@xxxxxxxxxxx > >>> http://www.gluster.org/mailman/listinfo/gluster-users > >>> > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users@xxxxxxxxxxx > > http://www.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
