On Fri, Dec 11, 2015 at 1:34 PM, John Stoffel <john@xxxxxxxxxxx> wrote: >>>>>> "Dallas" == Dallas Clement <dallas.a.clement@xxxxxxxxx> writes: > > Dallas> On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@xxxxxxxxxxx> wrote: >>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@xxxxxxxxx> writes: >>> > Dallas> Hi Mark. I have three different controllers on this > Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an > Dallas> Intel Cougar Point controls the 4 remaining disks. >>> >>> What type of PCIe slots are the controllers in? And how fast are the >>> controllers/drives? Are they SATA1/2/3 drives? >>> >>>>> If you're spinning in IO loops then it could be a driver issue. >>> > Dallas> It sure is looking like that. I will try to profile the > Dallas> kernel threads today and maybe use blktrace as Phil > Dallas> recommended to see what is going on there. >>> >>> what kernel aer you running? >>> > Dallas> This is pretty sad that 12 single threaded fio jobs can bring > Dallas> this system to its knees. >>> >>> I think it might be better to lower the queue depth, you might be just >>> blowing out the controller caches... hard to know. > > Dallas> Hi John. > >>> What type of PCIe slots are the controllers in? And how fast are the >>> controllers/drives? Are they SATA1/2/3 drives? > > Dallas> The MV 9485 controller is attached to an Intel Sandy Bridge > Dallas> via PCIe GEN2 x 8. This one controls 8 of the disks. The > Dallas> Intel Cougar Point is connected to the Intel Sandy Bridge via > Dallas> DMI bus. > > So that should all be nice and fast. > > Dallas> All of the drives are SATA III, however I do have two of the > Dallas> drives connected to SATA II ports on the Cougar Point. These > Dallas> two drives used to be connected to SATA III ports on a MV > Dallas> 9125/9120 controller. But it had truly horrible write > Dallas> performance. Moving to the SATA II ports on the Cougar Point > Dallas> boosted the performance close to the same as the other drives. > Dallas> The remaining 10 drives are all connected to SATA III ports. > >>> what kernel aer you running? > > Dallas> Right now, I'm using 3.10.69. But I have tried the 4.2 kernel > Dallas> in Fedora 23 with similar results. > > Hmm... maybe if your feeling adventerous you could try v4.4-rc4 and > see how it works. You don't want anything between 4.2.6 and that > because of problems with blk req management. I'm hazy on the details. > >>> I think it might be better to lower the queue depth, you might be just >>> blowing out the controller caches... hard to know. > > Dallas> Good idea. I'll trying lowering to see what effect. > > It might also make sense to try your tests starting with just 1 disk, > and then adding one more disk, re-running the tests, then another > disk, re-running the tests, etc. > > Try with one on the MV, then one on the Cougar, then one on MV and one > on Cougar, etc. > > Try to see if you can spot where the performance falls off the cliff. > > Also, which disk scheduler are you using? Instead of CFQ, you might > try deadline instead. > > As you can see, there's a TON of knobs to twiddle with, it's not a > simple thing to do at times. > > John > It might also make sense to try your tests starting with just 1 disk, > and then adding one more disk, re-running the tests, then another > disk, re-running the tests, etc > Try to see if you can spot where the performance falls off the cliff. Okay, did this. Interestingly, things did not fall of the cliff until adding in the 12th disk. I started adding disks one at a time beginning with the Cougar Point. The %iowait jumped up right away with this guy also. > Also, which disk scheduler are you using? Instead of CFQ, you might > try deadline instead. I'm using deadline. I have definitely observed better performance with this vs cfq. At this point I think I need to probably use a tool like blktrace to get more visibility than what I have with ps and iostat. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
