Hi Jens, On Mon, 12 Feb 2024 11:36:42 -0700, Jens Axboe <axboe@xxxxxxxxx> wrote: > > For the record, here is my set up and performance data for 4 Samsung > > disks. IOPS increased from 1.6M per disk to 2.1M. One difference I > > noticed is that IRQ throughput is improved instead of reduction with > > this patch on my setup. e.g. BEFORE: 185545/sec/vector > > AFTER: 220128 > > I'm surprised at the rates being that low, and if so, why the posted MSI > makes a difference? Usually what I've seen for IRQ being slower than > poll is if interrupt delivery is unreasonably slow on that architecture > of machine. But ~200k/sec isn't that high at all. Even at ~200k/sec, I am seeing around 75% ratio between posted interrupt notification and MSIs. i.e. for every 4 MSIs, we save one CPU notification. That might be where the savings come from. I was expecting an even or reduction in CPU notifications but more MSI throughput. Instead, Optane gets less MSIs/sec as your data shows. Is it possible to get the interrupt coalescing ratio on your set up? ie. PMN count in cat /proc/interrupts divided by total NVME MSIs. Here is a summary of my testing on 4 Samsung Gen 5 drives: test cases IOPS*1000 ints/sec(MSI)* ================================================= aio 6348 182218 io_uring 6895 207932 aio w/ posted MSI 8295 185545 io_uring w/ post MSI 8811 220128 io_uring poll_queue 13000 0 ================================================ Thanks, Jacob
