Hi, Paolo Valente and I found an unexpected surprise while investigating iodepth impact on throughput. Both on my device (ssd samsung 960 pro nvme m.2 512gb) and his one (ssd plextor px-256m5s), throughput decreases as the iodepth increases. What follows is a summary of the testbed details: - mq-deadline scheduler - 4 fio doing rand-read I/O (4 is the number of physical cores of our cpus) - throughput was measured with "iostat -tmd /dev/$disk 3" - from the above command the first "mb_read/s" occurrence was ignored because of ramp time and only on the following ones average and stdev were computed - duration of 15 sec More details can be found in the jobfiles in attachment, please note that: - global sections differ only for the first two lines (iodepth and ioengine) - job sections differ only for N (0 to 3) What we observed is that with "iodepth=1 and ioengine=sync" throughput is greater than with "iodepth=8 and ioengine=libaio". For example: - iodepth=1, sync: 170.942 ± 11.122 [mb/s] - iodepth=8, async: 161.360 ± 10.165 [mb/s] So, I would like to ask: 1) Is it normal that throughput drop? (shouldn't device internal queues be more filled with a greater iodepth and thus lead to a greater throughput?) 2) If (1) is normal, for which reasons? 3) If (1) is not normal, we thought to investigate about the average number of requests in software and hardware queues (especially the latter ones). In fact we expect a correlation between that number and throughput. 4) If (3) is correct, how can we measure the average number of requests in the device internal queues? Is it possible to do that directly with fio? 5) If (3) is not correct, what should we look for? and (if possible) with which tools? Under request I can give more details. Thank you, Federico
<<attachment: fio_jobfiles.zip>>
Attachment:
signature.asc
Description: OpenPGP digital signature
