On 16/11/2020 21:24, Bijan Mottahedeh wrote:
> On 11/15/2020 5:33 AM, Pavel Begunkov wrote:
>> On 12/11/2020 23:00, Bijan Mottahedeh wrote:
>>> Apply fixed_rsrc functionality for fixed buffers support.
>>
>> I don't get it, requests with fixed files take a ref to a node (see
>> fixed_file_refs) and put it on free, but I don't see anything similar
>> here. Did you work around it somehow?
>
> No that's my oversight. I think I wrongfully assumed that io_import_*fixed() would take care of that.
>
> Should I basically do something similar to io_file_get()/io_put_file()?
If done in a dumb way, that'd mean extra pair of percpu get/put
and +8B in io_kiocb. Frankly, I don't like that idea.
However, if you don't split paths and make fixed_file_ref_node to
supports all types of resources at the same time, it should be
bearable. I.e. register removals of both types to a single node,
and use ->fixed_file_refs for all request's resources.
So you don't grow io_kiocb and do maximum one percpu_ref_get/put()
pair per request.
I'll send a small patch preparing grounds, because there is actually
another nasty thing from past that needs to be reworked.
>
> io_import_fixed()
> io_import_iovec_fixed()
> -> io_buf_get()
>
> io_dismantle_io()
> -> io_put_buf()
>
>>
>> That's not critical for this particular patch as you still do full
>> quisce in __io_uring_register(), but IIRC was essential for
>> update/remove requests.
>
> That's something I'm not clear about. Currently we quiesce for the following cases:
>
> case IORING_UNREGISTER_FILES:
> case IORING_REGISTER_FILES_UPDATE:
> case IORING_REGISTER_BUFFERS_UPDATE:
static bool io_register_op_must_quiesce(int op)
{
switch (op) {
case IORING_UNREGISTER_FILES:
case IORING_REGISTER_FILES_UPDATE:
case IORING_REGISTER_PROBE:
case IORING_REGISTER_PERSONALITY:
case IORING_UNREGISTER_PERSONALITY:
return false;
default:
return true;
}
}
It returns _false_ for these cases, so _doesn't_ quiesce for them.
>
> I had assume I have to add IORING_UNREGISTER_BUFFERS as well. But above, do we in fact the quiesce give the ref counts?
>
> Are you ok with the rest of the patches or should I address anything else?
io_import_fixed() currently can be called twice, and that would give
you 2 different bvecs. Hence after removing full quisce io_read()
retrying short reads will probably be able to partially read into 2
different buffers. That really have to be fixed.
I haven't looked the patchset properly yet. I'll reply to the
cover-letter + a small comment below
>>> static ssize_t io_import_fixed(struct io_kiocb *req, int rw,
>>> struct iov_iter *iter)
>>> {
>>> @@ -2959,10 +2982,15 @@ static ssize_t io_import_fixed(struct io_kiocb *req, int rw,
>>> size_t offset;
>>> u64 buf_addr;
>>> + /* attempt to use fixed buffers without having provided iovecs */
>>> + if (unlikely(!ctx->buf_data))
>>> + return -EFAULT;
I removed it for files,
because (ctx->buf_data) IFF (ctx->nr_user_bufs == 0),
so the following ctx->nr_user_bufs check is enough.
>>> +
>>> + buf_index = req->buf_index;
>>> if (unlikely(buf_index >= ctx->nr_user_bufs))
>>> return -EFAULT;
>>> index = array_index_nospec(buf_index, ctx->nr_user_bufs);
>>> - imu = &ctx->user_bufs[index];
>>> + imu = io_buf_from_index(ctx, index);
>>> buf_addr = req->rw.addr;
>>> /* overflow */
>>> @@ -8167,28 +8195,73 @@ static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries)
>>> return pages;
>>> }
--
Pavel Begunkov
