On 2/9/21 5:03 PM, Pavel Begunkov wrote: > Unfolding previous ideas on persistent req caches. 4-7 including > slashed ~20% of overhead for nops benchmark, haven't done benchmarking > personally for this yet, but according to perf should be ~30-40% in > total. That's for IOPOLL + inline completion cases, obviously w/o > async/IRQ completions. And task_work, which is sort-of inline. > Jens, > 1. 11/17 removes deallocations on end of submit_sqes. Looks you > forgot or just didn't do that. > > 2. lists are slow and not great cache-wise, that why at I want at least > a combined approach from 12/17. This is only true if you're browsing a full list. If you do add-to-front for a cache, and remove-from-front, then cache footprint of lists are good. > 3. Instead of lists in "use persistent request cache" I had in mind a > slightly different way: to grow the req alloc cache to 32-128 (or hint > from the userspace), batch-alloc by 8 as before, and recycle _all_ reqs > right into it. If overflows, do kfree(). > It should give probabilistically high hit rate, amortising out most of > allocations. Pros: it doesn't grow ~infinitely as lists can. Cons: there > are always counter examples. But as I don't have numbers to back it, I > took your implementation. Maybe, we'll reconsider later. It shouldn't grow bigger than what was used, but the downside is that it will grow as big as the biggest usage ever. We could prune, if need be, of course. As far as I'm concerned, the hint from user space is the submit count. > I'll revise tomorrow on a fresh head + do some performance testing, > and is leaving it RFC until then. I'll look too and test this, thanks! -- Jens Axboe
