On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi <manish.awasthi@xxxxxxxxxxxxxxxxxx> wrote: > > On 11/25/2014 08:07 AM, NeilBrown wrote: > > On Mon, 24 Nov 2014 13:40:06 +0530 Manish Awasthi > > <manish.awasthi@xxxxxxxxxxxxxxxxxx> wrote: > > > >> Hi, > >> > >> We benchmarked the md raid driver performance on 3-18-rc3 kernel and > >> compared the results with that of 3.6.11. The reason for this exercise > >> is to understand if multithreaded raid driver has any performance > >> benefits over 3.6.11 which is single threaded. Here are some details > >> about the setup > > Thanks for doing this!!!! I love it when people report test results. > > > > > >> System: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz 4 cores (8threads), > >> 8GB RAM. > >> Setup: 3 SSDs create a raid5 array > >> test tool: iozone (only read/re-read, write/re-write tested), blocksize: > >> 4k-64k, filesize: 1Gig to 200Gig > >> > >> Comparison was done for speed of data transfer in kBytes/sec and also > >> the CPU utilization as reported by iozone. > >> > >> raid on 3.18.0-rc3 performed much worse than raid on 3.6.11. > >> > >> Read/Write: raid on 3.18.0-rc3 operated at almost half the speed of raid > >> on 3.6.11 > > That really isn't very good.... Can you try some of the kernels in between > > and see if there was a single point where performance dropped, or if there > > were several steps? > Can you give me some starting point when multithread support for raid > was added. It might be a good starting point. For instance, I have a > benchmark on 3.6.11, now I'd like to go to the first kernel that has > support for multithread raid and then take it from there. Multi-thread support appeared in 3.12. > > > > > >> CPU Utilization: With md raid on 3.18.0-rc3, the CPU utilization was > >> less than half of md raid on 3.6.11 on WRITE operations. However, for > >> READ operations, 3.18-0.rc3 had more CPU utilization than 3.6.11. > > Can you use "perf" to determine where the extra time is going? > > > > perf record > > run test > > stop perf > > perf report > > > > or something like that. > I can do this but as I mentioned below, Its better if I can get to > understand all the possible tweaks that can be done to get the optimal > results unless ofcourse you expect 3.18.0 to perform better than that of > 3.6.11 even if default case without any tweaks. I have no particular expectations. I like to see concrete measurements and then try to interpret them. I prefer to compare default setting (no tweaks) in the first instance. Because that is what most people will be using. > > > >> Also, I noticed that scaling up the CPU cores of the system scales down > >> the raid througput with 3.18.0-rc3. > > This is by writing numbers to "group_thread_cnt" ??? Can you provide a simple > > table comparing thread count to throughput? Or maybe a graph. I love > > graphs :-) > I did not tweak anything on the 3.18.0 kernel. I assumed all the > required support is built-in and did not bother to go into the depth of > the code as we're still in nascent stages where we are comparing the > data on specific kernel versions. Can you point me to some text that can > describe tweaks like "group_thread_cnt" etc? Multi-threading is disabled by default, so if you haven't explicitly enabled it, then it cannot be affecting your performance. If you echo 8 > /sys/block/mdXXX/md/group_thread_cnt it will use 8 thread to perform 'xor' calculations and submit IO requests. > > > >> I do have detailed logs of the comparison but I'm not sure I should send > >> those on this mailing list. > > A few megabytes? Yes. 100Meg? No. > > Whatever data I have on comparison is attached, I have consolidated this > from log files to excel. See if this helps. Thanks. I'll have a look at the tables and see if anything looks interesting. Thanks, NeilBrown > > > > If you could put them on a website somewhere that I can browse or download > > I'll try to have a look. > > > >> If my observation aligns with someone else's, then what is really the > >> gain with multithreaded raid. > > Some testing shows real improvements. Obviously we cannot test everything > > and I'm very glad to have extra testing from other people. > > If we can quantify the regressions and confirm exactly when they occurred, we > > can start looking for a solution. > > > > Thanks a lot! > > > > NeilBrown > > > > > >> Manish > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in > >> the body of a message to majordomo@xxxxxxxxxxxxxxx > >> More majordomo info at http://vger.kernel.org/majordomo-info.html >
Attachment:
pgpMfW3JYCeZ7.pgp
Description: OpenPGP digital signature
