One thing that's always been a bit slower than I'd like with io_uring is dealing with pipes. They don't support IOCB_NOWAIT, and hence we need to punt them to io-wq for handling. This series adds support for FMODE_NOWAIT to pipes. Patch 1 extends pipe_buf_operations->confirm() to accept a nonblock parameter, and wires up the caller, pipe_buf_confirm(), to have that argument too. Patch 2 makes pipes deal with IOCB_NOWAIT for locking the pipe, calling pipe_buf_confirm(), and for allocating new pages on writes. Patch 3 flicks the switch and enables FMODE_NOWAIT for pipes. Curious on how big of a difference this makes, I wrote a small benchmark that simply opens 128 pipes and then does 256 rounds of reading and writing to them. This was run 10 times, discarding the first run as it's always a bit slower. Before the patch: Avg: 262.52 msec Stdev: 2.12 msec Min: 261.07 msec Max 267.91 msec and after the patch: Avg: 24.14 msec Stdev: 9.61 msec Min: 17.84 msec Max: 43.75 msec or about a 10x improvement in performance (and efficiency) for pipes being empty on read attempt. If we run the same test but with pipes already having data, the improvement is even better (as expected): Before: Avg: 249.24 msec Stdev: 0.20 msec Min: 248.96 msec Max: 249.53 msec After: Avg: 10.86 msec Stdev: 0.91 msec Min: 10.02 msec Max: 12.67 msec or about a 23x improvement. I ran the patches through the ltp pipe and splice tests, no regressions observed. Looking at io_uring traces, we can see that we no longer have any io_uring_queue_async_work() traces after the patch, where previously everything was done via io-wq. Changes since v1: - Add acks/reviewed-bys - Fix missing __GFP_HARDWALL (willy) - Get rid of nasty double ternary (willy,christian) -- Jens Axboe
