Re: (subset) [PATCH 00/11] remove aux CQE caches

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 3/16/24 04:13, Pavel Begunkov wrote:
On 3/16/24 03:54, Ming Lei wrote:
On Sat, Mar 16, 2024 at 02:54:19AM +0000, Pavel Begunkov wrote:
On 3/16/24 02:24, Ming Lei wrote:
On Sat, Mar 16, 2024 at 10:04 AM Ming Lei <ming.lei@xxxxxxxxxx> wrote:

On Fri, Mar 15, 2024 at 04:53:21PM -0600, Jens Axboe wrote:

On Fri, 15 Mar 2024 15:29:50 +0000, Pavel Begunkov wrote:
Patch 1 is a fix.

Patches 2-7 are cleanups mainly dealing with issue_flags conversions,
misundertsandings of the flags and of the tw state. It'd be great to have
even without even w/o the rest.

8-11 mandate ctx locking for task_work and finally removes the CQE
caches, instead we post directly into the CQ. Note that the cache is
used by multishot auxiliary completions.

[...]

Applied, thanks!

Hi Jens and Pavel,

Looks this patch causes hang when running './check ublk/002' in blktests.

Not take close look, and  I guess it hangs in

io_uring_cmd_del_cancelable() -> io_ring_submit_lock

Thanks, the trace doesn't completely explains it, but my blind spot
was io_uring_cmd_done() potentially grabbing the mutex. They're
supposed to be irq safe mimicking io_req_task_work_add(), that's how
nvme passthrough uses it as well (but at least it doesn't need the
cancellation bits).

One option is to replace it with a spinlock, the other is to delay
the io_uring_cmd_del_cancelable() call to the task_work callback.
The latter would be cleaner and more preferable, but I'm lacking
context to tell if that would be correct. Ming, what do you think?

I prefer to the latter approach because the two cancelable helpers are
run in fast path.

Looks all new io_uring_cmd_complete() in ublk have this issue, and the
following patch should avoid them all.

The one I have in mind on top of the current tree would be like below.
Untested, and doesn't allow this cancellation thing for iopoll. I'll
prepare something tomorrow.


diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
index e45d4cd5ef82..000ba435451c 100644
--- a/io_uring/uring_cmd.c
+++ b/io_uring/uring_cmd.c
@@ -14,19 +14,15 @@
  #include "rsrc.h"
  #include "uring_cmd.h"

-static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd,
-        unsigned int issue_flags)
+static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd)
  {
      struct io_kiocb *req = cmd_to_io_kiocb(cmd);
-    struct io_ring_ctx *ctx = req->ctx;

      if (!(cmd->flags & IORING_URING_CMD_CANCELABLE))
          return;

      cmd->flags &= ~IORING_URING_CMD_CANCELABLE;
-    io_ring_submit_lock(ctx, issue_flags);
      hlist_del(&req->hash_node);
-    io_ring_submit_unlock(ctx, issue_flags);
  }

  /*
@@ -80,6 +76,15 @@ static inline void io_req_set_cqe32_extra(struct io_kiocb *req,
      req->big_cqe.extra2 = extra2;
  }

+static void io_req_task_cmd_complete(struct io_kiocb *req,
+                     struct io_tw_state *ts)
+{
+    struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);
+
+    io_uring_cmd_del_cancelable(ioucmd);
+    io_req_task_complete(req, ts);
+}
+
  /*
   * Called by consumers of io_uring_cmd, if they originally returned
   * -EIOCBQUEUED upon receiving the command.
@@ -89,8 +94,6 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2,
  {
      struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);

-    io_uring_cmd_del_cancelable(ioucmd, issue_flags);
-
      if (ret < 0)
          req_set_fail(req);

@@ -105,7 +108,7 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2,
              return;

Not very well thought through... Here should be a *del_cancelable call
as well

          io_req_complete_defer(req);
      } else {
-        req->io_task_work.func = io_req_task_complete;
+        req->io_task_work.func = io_req_task_cmd_complete;
          io_req_task_work_add(req);
      }
  }


--
Pavel Begunkov




[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux