----- Original Message ----- > From: "Raghavendra Gowdappa" <rgowdapp@xxxxxxxxxx> > To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx> > Sent: Monday, June 27, 2016 12:48:49 PM > Subject: Re: performance issues Manoj found in EC testing > > > > ----- Original Message ----- > > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > To: "Xavier Hernandez" <xhernandez@xxxxxxxxxx> > > Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx> > > Sent: Monday, June 27, 2016 12:42:35 PM > > Subject: Re: performance issues Manoj found in EC testing > > > > > > > > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < xhernandez@xxxxxxxxxx > > > > > wrote: > > > > > > Hi Manoj, > > > > I always enable client-io-threads option for disperse volumes. It improves > > performance sensibly, most probably because of the problem you have > > detected. > > > > I don't see any other way to solve that problem. > > > > I agree. Updated the bug with same info. > > > > > > > > I think it would be a lot better to have a true thread pool (and maybe an > > I/O > > thread pool shared by fuse, client and server xlators) in libglusterfs > > instead of the io-threads xlator. This would allow each xlator to decide > > when and what should be parallelized in a more intelligent way, since > > basing > > the decision solely on the fop type seems too simplistic to me. > > > > In the specific case of EC, there are a lot of operations to perform for a > > single high level fop, and not all of them require the same priority. Also > > some of them could be executed in parallel instead of sequentially. > > > > I think it is high time we actually schedule(for which release) to get this > > in gluster. May be you should send out a doc where we can work out details? > > I will be happy to explore options to integrate io-threads, syncop/barrier > > with this infra based on the design may be. > > +1. I can volunteer too. Thanks, folks! As a quick update, throughput on a single client test jumped from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is now more in line with what is expected for this workload based on back-of-the-envelope calculations. Are there any reservations about recommending client-io-threads=on as "default" tuning, until the enhancement discussed above becomes reality? -- Manoj > > > > > > > > > Xavi > > > > > > On 25/06/16 19:42, Manoj Pillai wrote: > > > > > > > > ----- Original Message ----- > > > > > > From: "Pranith Kumar Karampuri" < pkarampu@xxxxxxxxxx > > > To: "Xavier Hernandez" < xhernandez@xxxxxxxxxx > > > Cc: "Manoj Pillai" < mpillai@xxxxxxxxxx >, "Gluster Devel" < > > gluster-devel@xxxxxxxxxxx > > > Sent: Thursday, June 23, 2016 8:50:44 PM > > Subject: performance issues Manoj found in EC testing > > > > hi Xavi, > > Meet Manoj from performance team Redhat. He has been testing EC > > performance in his stretch clusters. He found some interesting things we > > would like to share with you. > > > > 1) When we perform multiple streams of big file writes(12 parallel dds I > > think) he found one thread to be always hot (99%CPU always). He was asking > > me if fuse_reader thread does any extra processing in EC compared to > > replicate. Initially I thought it would just lock and epoll threads will > > perform the encoding but later realized that once we have the lock and > > version details, next writes on the file would be encoded in the same > > thread that comes to EC. write-behind could play a role and make the writes > > come to EC in an epoll thread but we saw consistently there was just one > > thread that is hot. Not multiple threads. We will be able to confirm this > > in tomorrow's testing. > > > > 2) This is one more thing Raghavendra G found, that our current > > implementation of epoll doesn't let other epoll threads pick messages from > > a socket while one thread is processing one message from that socket. In > > EC's case that can be encoding of the write/decoding read. This will not > > let replies of operations on different files to be processed in parallel. > > He thinks this can be fixed for 3.9. > > > > Manoj will be raising a bug to gather all his findings. I just wanted to > > introduce him and let you know the interesting things he is finding before > > you see the bug :-). > > -- > > Pranith > > > > Thanks, Pranith :). > > > > Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953 > > > > Comparing EC and replica-2 runs, the hot thread is seen in both cases, so > > I have not opened this as an EC bug. But initial impression is that > > performance impact for EC is particularly bad (details in the bug). > > > > -- Manoj > > > > > > > > > > -- > > Pranith > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@xxxxxxxxxxx > > http://www.gluster.org/mailman/listinfo/gluster-devel > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-devel > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
