Re: [REGRESSION] lxc-stop hang on 5.17.x kernels

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 5/2/22 19:49, Daniel Harding wrote:
On 5/2/22 20:40, Pavel Begunkov wrote:
On 5/2/22 18:00, Jens Axboe wrote:
On 5/2/22 7:59 AM, Jens Axboe wrote:
On 5/2/22 7:36 AM, Daniel Harding wrote:
On 5/2/22 16:26, Jens Axboe wrote:
On 5/2/22 7:17 AM, Daniel Harding wrote:
I use lxc-4.0.12 on Gentoo, built with io-uring support
(--enable-liburing), targeting liburing-2.1.  My kernel config is a
very lightly modified version of Fedora's generic kernel config. After
moving from the 5.16.x series to the 5.17.x kernel series, I started
noticed frequent hangs in lxc-stop.  It doesn't happen 100% of the
time, but definitely more than 50% of the time. Bisecting narrowed
down the issue to commit aa43477b040251f451db0d844073ac00a8ab66ee:
io_uring: poll rework. Testing indicates the problem is still present
in 5.18-rc5. Unfortunately I do not have the expertise with the
codebases of either lxc or io-uring to try to debug the problem
further on my own, but I can easily apply patches to any of the
involved components (lxc, liburing, kernel) and rebuild for testing or
validation.  I am also happy to provide any further information that
would be helpful with reproducing or debugging the problem.
Do you have a recipe to reproduce the hang? That would make it
significantly easier to figure out.

I can reproduce it with just the following:

     sudo lxc-create --n lxc-test --template download --bdev dir --dir /var/lib/lxc/lxc-test/rootfs -- -d ubuntu -r bionic -a amd64
     sudo lxc-start -n lxc-test
     sudo lxc-stop -n lxc-test

The lxc-stop command never exits and the container continues running.
If that isn't sufficient to reproduce, please let me know.

Thanks, that's useful! I'm at a conference this week and hence have
limited amount of time to debug, hopefully Pavel has time to take a look
at this.

Didn't manage to reproduce. Can you try, on both the good and bad
kernel, to do:

Same here, it doesn't reproduce for me
OK, sorry it wasn't something simple.
# echo 1 > /sys/kernel/debug/tracing/events/io_uring/enable

run lxc-stop

# cp /sys/kernel/debug/tracing/trace ~/iou-trace

so we can see what's going on? Looking at the source, lxc is just using
plain POLL_ADD, so I'm guessing it's not getting a notification when it
expects to, or it's POLL_REMOVE not doing its job. If we have a trace
from both a working and broken kernel, that might shed some light on it.
It's late in my timezone, but I'll try to work on getting those traces tomorrow.

I think I got it, I've attached a trace.

What's interesting is that it issues a multi shot poll but I don't
see any kind of cancellation, neither cancel requests nor task/ring
exit. Perhaps have to go look at lxc to see how it's supposed
to work

--
Pavel Begunkov
# tracer: nop
#
# entries-in-buffer/entries-written: 207/207   #P:16
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
       lxc-start-3026    [010] .....    58.519361: io_uring_create: ring 00000000de7fa538, fd 3 sq size 512, cq size 1024, flags 0x0
       lxc-start-3026    [010] .....    58.519418: io_uring_create: ring 0000000001d3ba30, fd 4 sq size 512, cq size 1024, flags 0x0
       lxc-start-3026    [010] .....    58.519433: io_uring_submit_sqe: ring 00000000de7fa538, req 0000000061bba231, user_data 0x5631784e5920, opcode 6, flags 0x80000, non block 1, sq_thread 0
       lxc-start-3026    [010] .....    58.519434: io_uring_file_get: ring 00000000de7fa538, req 0000000061bba231, user_data 0x5631784e5920, fd 7
       lxc-start-3026    [010] .....    58.519438: io_uring_submit_sqe: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, flags 0x80000, non block 1, sq_thread 0
       lxc-start-3026    [010] .....    58.519438: io_uring_file_get: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, fd 31
       lxc-start-3026    [010] .....    58.519442: io_uring_submit_sqe: ring 00000000de7fa538, req 000000004b8dead2, user_data 0x5631784e59c0, opcode 6, flags 0x80000, non block 1, sq_thread 0
       lxc-start-3026    [010] .....    58.519442: io_uring_file_get: ring 00000000de7fa538, req 000000004b8dead2, user_data 0x5631784e59c0, fd 5
       lxc-start-3026    [010] .....    58.519444: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:4-164     [005] d..1.    58.542112: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.542147: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.542165: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:4-164     [005] d..1.    58.542245: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.542272: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.542278: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:4-164     [005] d..1.    58.542419: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [002] ...1.    58.542433: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [002] .....    58.542449: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.705795: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [004] ...1.    58.705838: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [004] .....    58.705859: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.705947: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [004] ...1.    58.705981: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [004] .....    58.705990: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.706420: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.706460: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.706472: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.706516: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.706552: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.706561: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.706878: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.706910: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.706920: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.706932: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.706940: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.706942: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.707011: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.707044: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.707054: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.707527: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.707560: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.707569: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.708337: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.708360: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.708366: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.708443: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.708466: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.708472: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.708515: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.708538: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.708544: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.708597: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.708620: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.708626: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.708637: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [006] ...1.    58.708643: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [006] .....    58.708645: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.709330: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.709371: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.709387: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:4-164     [005] d..1.    58.711588: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.711605: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.711618: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.713658: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.713677: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.713697: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.716828: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.716849: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.716872: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [014] d..1.    58.717675: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.717693: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.717703: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.726493: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.726510: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.726526: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.727418: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.727436: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.727453: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [014] d..1.    58.730047: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.730076: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.730094: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [003] d..1.    58.730287: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.730309: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.730313: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [003] d..1.    58.730948: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [010] ...1.    58.730964: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [010] .....    58.730978: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [007] d..1.    58.734533: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.734563: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.734578: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [007] d..1.    58.737818: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.737838: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.737850: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [014] d..1.    58.739063: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.739102: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.739116: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.739723: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.739750: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.739758: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.750710: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [008] ...1.    58.750744: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [008] .....    58.750756: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.751685: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.751705: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.751717: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.751740: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.751749: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.751752: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.757615: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.757631: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.757646: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.757727: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.757735: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.757742: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.757998: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758009: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758015: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.758181: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758206: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758212: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.758275: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758299: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758305: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.758336: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758343: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758345: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [014] d..1.    58.758348: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758352: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758353: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.758365: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758372: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758373: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [012] d..1.    58.758937: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.758955: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.758961: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.769910: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.769952: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.769962: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.770059: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.770094: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.770103: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.770736: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.770757: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.770767: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.770876: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.770892: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.770896: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [002] d..1.    58.771562: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.771579: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.771586: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [004] d..1.    58.772074: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.772092: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.772098: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.772902: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.772924: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.772943: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.774121: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.774142: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.774157: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.774821: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.774835: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.774844: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.775258: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.775270: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.775276: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.775613: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.775622: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.775625: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.775706: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.775716: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.775717: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.776105: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.776114: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.776116: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:2-120     [005] d..1.    58.776501: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.776515: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.776518: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [002] d..1.    58.778234: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.778256: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.778278: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.779513: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.779540: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.779557: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.851129: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.851159: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.851182: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.851306: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.851318: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.851323: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.851347: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.851357: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.851361: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.852176: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.852212: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.852221: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [004] d..1.    58.859578: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    58.859605: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    58.859630: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [014] d..1.    59.861798: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    59.861827: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
       lxc-start-3026    [015] .....    59.861855: io_uring_cqring_wait: ring 00000000de7fa538, min_events 1
   kworker/u32:3-139     [014] d..1.    59.863176: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    59.863191: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
   kworker/u32:2-120     [005] d..1.    59.863195: io_uring_task_add: ring 00000000de7fa538, req 000000000dd2a118, user_data 0x5631784e5970, opcode 6, mask 41
       lxc-start-3026    [015] ...1.    59.863206: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e5970, result 1, cflags 0x2
        lxc-stop-3305    [005] d..1.    76.843006: io_uring_task_add: ring 00000000de7fa538, req 000000004b8dead2, user_data 0x5631784e59c0, opcode 6, mask c3
       lxc-start-3026    [015] ...1.    76.843057: io_uring_complete: ring 00000000de7fa538, req 0000000000000000, user_data 0x5631784e59c0, result 1, cflags 0x2

[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux