bio: convert get_user_pages_fast() --> pin_user_pages_fast()
Change generic block/bio Direct IO routines, to acquire FOLL_PIN user
pages via the recently added routines:
iov_iter_pin_pages()
iov_iter_pin_pages_alloc()
pin_page()
This effectively converts several file systems (ext4, for example) that
use the common Direct IO routines.
Change the corresponding page release calls from put_page() to
unpin_user_page().
Change bio_release_pages() to handle FOLL_PIN pages. In fact, after this
patch, that is the *only* type of pages that bio_release_pages()
handles.
Design notes
============
Quite a few approaches have been considered over the years. This one is
inspired by Christoph Hellwig's July, 2019 observation that there are
only 5 ITER_ types, and we can simplify handling of them for Direct IO
[1]. Accordingly, this patch implements the following pseudocode:
Direct IO behavior:
ITER_IOVEC:
pin_user_pages_fast();
break;
ITER_PIPE:
for each page:
pin_page();
break;
ITER_KVEC: // already elevated page refcount, leave alone
ITER_BVEC: // already elevated page refcount, leave alone
ITER_DISCARD: // discard
return -EFAULT or -ENVALID;
...which works for callers that already have sorted out which case they
are in. Such as, Direct IO in the block/bio layers.
Note that this does leave ITER_KVEC and ITER_BVEC unconverted, for now.
Page acquisition: The iov_iter_get_pages*() routines above are at just
the right level in the call stack: the callers already know which system
to use, and so it's a small change to just drop in the replacement
routines. And it's a fan-in/fan-out point: block/bio call sites for
Direct IO funnel their page acquisitions through the
iov_iter_get_pages*() routines, and there are many other callers of
those. And we can't convert all of the callers at once--too many
subsystems are involved, and it would be a too large and too risky
patch.
Page release: there are already separate release routines: put_page()
vs. unpin_user_page(), so it's already done there.
[1] https://lore.kernel.org/kvm/20190724061750.GA19397@xxxxxxxxxxxxx/
[2] "Explicit pinning of user-space pages":
https://lwn.net/Articles/807108/
John Hubbard (3):
mm/gup: introduce pin_page()
iov_iter: introduce iov_iter_pin_pages*() routines
bio: convert get_user_pages_fast() --> pin_user_pages_fast()
block/bio.c | 24 ++++-----
block/blk-map.c | 6 +--
fs/direct-io.c | 28 +++++------
fs/iomap/direct-io.c | 2 +-
include/linux/mm.h | 2 +
include/linux/uio.h | 5 ++
lib/iov_iter.c | 113 ++++++++++++++++++++++++++++++++++++++++---
mm/gup.c | 33 +++++++++++++
8 files changed, 175 insertions(+), 38 deletions(-)
--
2.28.0
