On 22/12/2020 11:04, Dmitry Kadashev wrote: > On Tue, Dec 22, 2020 at 11:11 AM Pavel Begunkov <asml.silence@xxxxxxxxx> wrote: [...] >>> What about smaller rings? Can you check io_uring of what SQ size it can allocate? >>> That can be a different program, e.g. modify a bit liburing/test/nop. > Unfortunately I've rebooted the box I've used for tests yesterday, so I can't > try this there. Also I was not able to come up with an isolated reproducer for > this yet. > > The good news is I've found a relatively easy way to provoke this on a test VM > using our software. Our app runs with "admin" user perms (plus some > capabilities), it bumps RLIMIT_MEMLOCK to infinity on start. I've also created > an user called 'ioutest' to run the check for ring sizes using a different user. > > I've modified the test program slightly, to show the number of rings > successfully > created on each iteration and the actual error message (to debug a problem I was > having with it, but I've kept this after that). Here is the output: > > # sudo -u admin bash -c 'ulimit -a' | grep locked > max locked memory (kbytes, -l) 1024 > > # sudo -u ioutest bash -c 'ulimit -a' | grep locked > max locked memory (kbytes, -l) 1024 > > # sudo -u admin ./iou-test1 > Failed after 0 rings with 1024 size: Cannot allocate memory > Failed after 0 rings with 512 size: Cannot allocate memory > Failed after 0 rings with 256 size: Cannot allocate memory > Failed after 0 rings with 128 size: Cannot allocate memory > Failed after 0 rings with 64 size: Cannot allocate memory > Failed after 0 rings with 32 size: Cannot allocate memory > Failed after 0 rings with 16 size: Cannot allocate memory > Failed after 0 rings with 8 size: Cannot allocate memory > Failed after 0 rings with 4 size: Cannot allocate memory > Failed after 0 rings with 2 size: Cannot allocate memory > can't allocate 1 > > # sudo -u ioutest ./iou-test1 > max size 1024 Then we screw that specific user. Interestingly, if it has CAP_IPC_LOCK capability we don't even account locked memory. btw, do you use registered buffers? > > # ps ax | grep wq > 8 ? I< 0:00 [mm_percpu_wq] > 121 ? I< 0:00 [tpm_dev_wq] > 124 ? I< 0:00 [devfreq_wq] > 20593 pts/1 S+ 0:00 grep --color=auto wq > -- Pavel Begunkov
