Re: Question: t/io_uring performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Jens,

thank you for your reply. Given that you have read the thread after the first reply, I think some of the questions of your first email are no longer relevant. I still answered them at the bottom for completeness, but I will answer the more interesting ones first.

I turn off iostats and merging for the device.



Doing this helped quite a bit. The 512b reads went from 715K to 800K. The 4096b reads went from 570K to 630K.

Note that you'll need to configure NVMe
 to properly use polling. I use 32 poll queues, number isn't really
 that important for single core testing, as long as there's enough to
 have a poll queue local to CPU being tested on.

My SSD was configured to use 128/0/0 default/read/poll queues. I added "nvme.poll_queues=32" to GRUB and rebooted, which changed it to 96/0/32. I now get 1.0M IOPS (512b blocks) and 790K IOPS (4096b blocks) using a single core. Thank you very much, this probably was the main bottleneck. Launching the benchmark two times with 512b blocks, I get 1.4M IOPS total.

Starting single-threaded t/io_uring with two SSDs still achieves "only" 1.0M IOPS, independently of the block size. In your benchmarks from 2019 [0] when Linux 5.4 (which I am using) was current, you achieved 1.6M IOPS (4096b blocks) using a single core. I get the full 1.6M IOPS for saturating both SSDs (4096b blocks) only when running t/io_uring with two threads. This makes me think that there is still another configuration option that I am missing. Most time is spent in the kernel.

# time taskset -c 48 t/io_uring -b512 -d128 -c32 -s32 -p1 -F1 -B1 /dev/nvme0n1 /dev/nvme1n1
i 8, argc 10
Added file /dev/nvme0n1 (submitter 0)
Added file /dev/nvme1n1 (submitter 0)
sq_ring ptr = 0x0x7f78fb740000
sqes ptr    = 0x0x7f78fb73e000
cq_ring ptr = 0x0x7f78fb73c000
polled=1, fixedbufs=1, register_files=1, buffered=0 QD=128, sq_ring=128, cq_ring=256
submitter=2336
IOPS=1014252, IOS/call=31/31, inflight=102 (38, 64)
IOPS=1017984, IOS/call=31/31, inflight=123 (64, 59)
IOPS=1018220, IOS/call=31/31, inflight=102 (38, 64)
[...]
real    0m7.898s
user    0m0.144s
sys     0m7.661s

I attached a perf output to the email. It was generated using the same parameters as above (getting 1.0M IOPS).

Thank you very much for your help. I am looking forward to hearing from you again to be able fully reproduce your measurements soon.
Hans-Peter


=== Answers to (I think) no longer relevant questions ===

The options I run t/io_uring with have been posted multiple times, it's this one

This is the same configuration that I also ran (I did not explicitly specify the parameters that are the same as the default).

Make sure your nvme device is using 'none' as the IO scheduler.

The scheduler is set to 'none'.

Is this a gen2 optane?

It is not an optane disk but I also do not expect to get insanely high numbers like in your recent benchmarks. Just more close to the old benchmarks but using two SSDs.


=== References ===

[0]: https://twitter.com/axboe/status/1174777844313911296

Attachment: perf-output.gz
Description: application/gzip


[Index of Archives]     [Linux Kernel]     [Linux SCSI]     [Linux IDE]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux SCSI]

  Powered by Linux