On Thu, Mar 30, 2023 at 10:33 AM Jens Axboe <axboe@xxxxxxxxx> wrote:
>
> That said, there might be things to improve here. But that's a task
> for another time.
So I ended up looking at this, and funnily enough, the *compat*
version of the "copy iovec from user" is actually written to be a lot
more efficient than the "native" version.
The reason is that the compat version has to load the data one field
at a time anyway to do the conversion, so it open-codes the loop. And
it does it all using the efficient "user_access_begin()" etc, so it
generates good code.
In contrast, the native version just does a "copy_from_user()" and
then loops over the result to verify it. And that's actually pretty
horrid. Doing the open-coded loop that fetches and verifies the iov
entries one at a time should be much better.
I dunno. That's my gut feel, at least. And it may explain why your
"readv()" benchmark has "_copy_from_user()" much higher up than the
"read()" case.
Something like the attached *may* help.
Untested - I only checked the generated assembly to see that it seems
to be sane, but I might have done something stupid. I basically copied
the compat code, fixed it up for non-compat types, and then massaged
it a bit more.
Linus
lib/iov_iter.c | 35 ++++++++++++++++++++++++++---------
1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 274014e4eafe..e793d6ca299c 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1731,18 +1731,35 @@ static int copy_compat_iovec_from_user(struct iovec *iov,
}
static int copy_iovec_from_user(struct iovec *iov,
- const struct iovec __user *uvec, unsigned long nr_segs)
+ const struct iovec __user *uiov, unsigned long nr_segs)
{
- unsigned long seg;
+ int ret = -EFAULT;
- if (copy_from_user(iov, uvec, nr_segs * sizeof(*uvec)))
+ if (!user_access_begin(uiov, nr_segs * sizeof(*uiov)))
return -EFAULT;
- for (seg = 0; seg < nr_segs; seg++) {
- if ((ssize_t)iov[seg].iov_len < 0)
- return -EINVAL;
- }
- return 0;
+ do {
+ void __user *buf;
+ ssize_t len;
+
+ unsafe_get_user(len, &uiov->iov_len, uaccess_end);
+ unsafe_get_user(buf, &uiov->iov_base, uaccess_end);
+
+ /* check for size_t not fitting in ssize_t .. */
+ if (unlikely(len < 0)) {
+ ret = -EINVAL;
+ goto uaccess_end;
+ }
+ iov->iov_base = buf;
+ iov->iov_len = len;
+
+ uiov++; iov++;
+ } while (--nr_segs);
+
+ ret = 0;
+uaccess_end:
+ user_access_end();
+ return ret;
}
struct iovec *iovec_from_user(const struct iovec __user *uvec,
@@ -1767,7 +1784,7 @@ struct iovec *iovec_from_user(const struct iovec __user *uvec,
return ERR_PTR(-ENOMEM);
}
- if (compat)
+ if (unlikely(compat))
ret = copy_compat_iovec_from_user(iov, uvec, nr_segs);
else
ret = copy_iovec_from_user(iov, uvec, nr_segs);
