Thanks for the response. It helps a little by increasing group_thread_cnt, but not to the extend of 3x expected. It looks like the single kernel thread is still the bottleneck. On Tue, Jan 17, 2017 at 12:04 AM, Coly Li <colyli@xxxxxxx> wrote: > On 2017/1/17 上午11:10, Stan Hoeppner wrote: >> On 01/16/2017 08:35 PM, Jake Yao wrote: >>> I have a raid5 array on 4 NVMe drives, and the performance on the >>> array is only marginally better than a single drive. Unlike a similar >>> raid5 array on 4 SAS SSD or HDD, the performance on array is 3x >>> better than a single drive, which is expected. >>> >>> It looks like when the single kernel thread associated with the raid >>> device running at 100%, the array performance hit its peak. This can >>> happen easily for fast devices like NVMe. >> The md raid personalities are limited to a single kernel write thread. >> Work is in progress to alleviate this bottleneck by using multiple write >> threads. When it will hit mainline I don't know. > > If you want 8 writing threads, and your md raid5 device is /dev/md0, you > may have a try with, > echo 8 > /sys/block/md0/md/group_thread_cnt > >> >>> This can reproduced by creating a raid5 with 4 ramdisks as well, and >>> comparing performance on the array and one ramdisk. Sometimes the >>> performance on the array is worse than a single ramdisk. >>> >>> The kernel version is 4.9.0-rc3 and mdadm is release 3.4, no write >>> journal is configured. >>> >>> Is this a known issue? > > It was, but you are on 4.9 kernel, group_thread_cnt should work for you. > > Coly > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
