On 12/11/2015 07:00 PM, Dallas Clement wrote: >> So is my workload of 12 fio jobs writing sequential 2 MB blocks with >> direct I/O just too abusive? Seems so with high queue depth. I don't think you are adjusting any hardware queue depth here. The fio man page is quite explicit that iodepth=N is ineffective for sequential operations. But you are using the libaio engine, so you are piling up many *software* queued operations for the kernel to execute, not operations in flight to the disks. From the histograms in your results, the vast majority of ops are completing at depth=4. Further queuing is just adding kernel overhead. The queuing differences from one kernel to another is a driver and hardware property, not an application property. >> I started this discussion because my RAID 5 and RAID 6 write >> performance is really bad. If my system is able to write to all 12 >> disks at 170 MB/s in JBOD mode, I am expecting that one fio job should >> be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870 >> MB/s. However, I am getting < 700 MB/s for queue depth = 32 and < 600 >> MB/s for queue depth = 256. I get similarly disappointing results for >> RAID 6 writes. That's why I suggested blktrace. Collect a trace while a single dd is writing to your raw array device. Compare the large writes submitted to the md device against the broken down writes submitted to the member devices. Compare the patterns and sizes from older kernels against newer kernels, possibly varying which controllers and data paths are involved. Phil -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
