----- Original Message ----- > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > To: "Xavier Hernandez" <xhernandez@xxxxxxxxxx> > Cc: "Manoj Pillai" <mpillai@xxxxxxxxxx>, "Gluster Devel" <gluster-devel@xxxxxxxxxxx> > Sent: Thursday, June 23, 2016 8:50:44 PM > Subject: performance issues Manoj found in EC testing > > hi Xavi, > Meet Manoj from performance team Redhat. He has been testing EC > performance in his stretch clusters. He found some interesting things we > would like to share with you. > > 1) When we perform multiple streams of big file writes(12 parallel dds I > think) he found one thread to be always hot (99%CPU always). He was asking > me if fuse_reader thread does any extra processing in EC compared to > replicate. Initially I thought it would just lock and epoll threads will > perform the encoding but later realized that once we have the lock and > version details, next writes on the file would be encoded in the same > thread that comes to EC. write-behind could play a role and make the writes > come to EC in an epoll thread but we saw consistently there was just one > thread that is hot. Not multiple threads. We will be able to confirm this > in tomorrow's testing. > > 2) This is one more thing Raghavendra G found, that our current > implementation of epoll doesn't let other epoll threads pick messages from > a socket while one thread is processing one message from that socket. In > EC's case that can be encoding of the write/decoding read. This will not > let replies of operations on different files to be processed in parallel. > He thinks this can be fixed for 3.9. > > Manoj will be raising a bug to gather all his findings. I just wanted to > introduce him and let you know the interesting things he is finding before > you see the bug :-). > -- > Pranith Thanks, Pranith :). Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953 Comparing EC and replica-2 runs, the hot thread is seen in both cases, so I have not opened this as an EC bug. But initial impression is that performance impact for EC is particularly bad (details in the bug). -- Manoj _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
