On Sat, Mar 16, 2024 at 04:20:25AM +0000, Pavel Begunkov wrote:
> On 3/16/24 04:13, Pavel Begunkov wrote:
> > On 3/16/24 03:54, Ming Lei wrote:
> > > On Sat, Mar 16, 2024 at 02:54:19AM +0000, Pavel Begunkov wrote:
> > > > On 3/16/24 02:24, Ming Lei wrote:
> > > > > On Sat, Mar 16, 2024 at 10:04 AM Ming Lei <ming.lei@xxxxxxxxxx> wrote:
> > > > > >
> > > > > > On Fri, Mar 15, 2024 at 04:53:21PM -0600, Jens Axboe wrote:
> > > > > > >
> > > > > > > On Fri, 15 Mar 2024 15:29:50 +0000, Pavel Begunkov wrote:
> > > > > > > > Patch 1 is a fix.
> > > > > > > >
> > > > > > > > Patches 2-7 are cleanups mainly dealing with issue_flags conversions,
> > > > > > > > misundertsandings of the flags and of the tw state. It'd be great to have
> > > > > > > > even without even w/o the rest.
> > > > > > > >
> > > > > > > > 8-11 mandate ctx locking for task_work and finally removes the CQE
> > > > > > > > caches, instead we post directly into the CQ. Note that the cache is
> > > > > > > > used by multishot auxiliary completions.
> > > > > > > >
> > > > > > > > [...]
> > > > > > >
> > > > > > > Applied, thanks!
> > > > > >
> > > > > > Hi Jens and Pavel,
> > > > > >
> > > > > > Looks this patch causes hang when running './check ublk/002' in blktests.
> > > > >
> > > > > Not take close look, and I guess it hangs in
> > > > >
> > > > > io_uring_cmd_del_cancelable() -> io_ring_submit_lock
> > > >
> > > > Thanks, the trace doesn't completely explains it, but my blind spot
> > > > was io_uring_cmd_done() potentially grabbing the mutex. They're
> > > > supposed to be irq safe mimicking io_req_task_work_add(), that's how
> > > > nvme passthrough uses it as well (but at least it doesn't need the
> > > > cancellation bits).
> > > >
> > > > One option is to replace it with a spinlock, the other is to delay
> > > > the io_uring_cmd_del_cancelable() call to the task_work callback.
> > > > The latter would be cleaner and more preferable, but I'm lacking
> > > > context to tell if that would be correct. Ming, what do you think?
> > >
> > > I prefer to the latter approach because the two cancelable helpers are
> > > run in fast path.
> > >
> > > Looks all new io_uring_cmd_complete() in ublk have this issue, and the
> > > following patch should avoid them all.
> >
> > The one I have in mind on top of the current tree would be like below.
> > Untested, and doesn't allow this cancellation thing for iopoll. I'll
> > prepare something tomorrow.
> >
> >
> > diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
> > index e45d4cd5ef82..000ba435451c 100644
> > --- a/io_uring/uring_cmd.c
> > +++ b/io_uring/uring_cmd.c
> > @@ -14,19 +14,15 @@
> > #include "rsrc.h"
> > #include "uring_cmd.h"
> >
> > -static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd,
> > - unsigned int issue_flags)
> > +static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd)
> > {
> > struct io_kiocb *req = cmd_to_io_kiocb(cmd);
> > - struct io_ring_ctx *ctx = req->ctx;
> >
> > if (!(cmd->flags & IORING_URING_CMD_CANCELABLE))
> > return;
> >
> > cmd->flags &= ~IORING_URING_CMD_CANCELABLE;
> > - io_ring_submit_lock(ctx, issue_flags);
> > hlist_del(&req->hash_node);
> > - io_ring_submit_unlock(ctx, issue_flags);
> > }
> >
> > /*
> > @@ -80,6 +76,15 @@ static inline void io_req_set_cqe32_extra(struct io_kiocb *req,
> > req->big_cqe.extra2 = extra2;
> > }
> >
> > +static void io_req_task_cmd_complete(struct io_kiocb *req,
> > + struct io_tw_state *ts)
> > +{
> > + struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);
> > +
> > + io_uring_cmd_del_cancelable(ioucmd);
> > + io_req_task_complete(req, ts);
> > +}
> > +
> > /*
> > * Called by consumers of io_uring_cmd, if they originally returned
> > * -EIOCBQUEUED upon receiving the command.
> > @@ -89,8 +94,6 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2,
> > {
> > struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
> >
> > - io_uring_cmd_del_cancelable(ioucmd, issue_flags);
> > -
> > if (ret < 0)
> > req_set_fail(req);
> >
> > @@ -105,7 +108,7 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2,
> > return;
>
> Not very well thought through... Here should be a *del_cancelable call
> as well
Thanks for the fix!
The patch works after adding io_uring_cmd_del_cancelable() in the branch of
`else if (issue_flags & IO_URING_F_COMPLETE_DEFER)'.
Thanks,
Ming
