There are GUP references to pages that are serving as direct IO buffers.
Those pages can be allocated from CMA pageblocks despite they can be
pinned until the DIO is completed.
Generally, pinning for each DIO might be considered as a transient
operation as described at the documentation. But if a large amount of
direct IO is requested constantly, this can make pages in CMA pageblocks
pinned and unable to migrate outside of the pageblock, which can result
in CMA allocation failure.
In Android devices, on first boot after OTA, snapuserd requests a huge
amount of direct IO reads which might occasionally disturb CMA
allocations.
To prevent this, use FOLL_LONGTERM as gup_flags for direct IO requests
via blkdev_direct_IO or __iomap_dio_rw by default not to allocate buffer
pages from CMA pageblocks.
Signed-off-by: Sooyong Suk <s.suk@xxxxxxxxxxx>
---
block/bio.c | 2 +-
include/linux/uio.h | 2 ++
lib/iov_iter.c | 2 ++
3 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/block/bio.c b/block/bio.c
index d5bdc31d88d3..683113b3e35a 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1247,7 +1247,7 @@ static unsigned int get_contig_folio_len(unsigned int *num_pages,
*/
static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter)
{
- iov_iter_extraction_t extraction_flags = 0;
+ iov_iter_extraction_t extraction_flags = ITER_ALLOW_LONGTERM;
unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt;
unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt;
struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt;
diff --git a/include/linux/uio.h b/include/linux/uio.h
index 853f9de5aa05..d1e9174ee29a 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -377,6 +377,8 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction,
/* Flags for iov_iter_get/extract_pages*() */
/* Allow P2PDMA on the extracted pages */
#define ITER_ALLOW_P2PDMA ((__force iov_iter_extraction_t)0x01)
+/* Allow LONGTERM on the extracted pages */
+#define ITER_ALLOW_LONGTERM ((__force iov_iter_extraction_t)0x02)
ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages,
size_t maxsize, unsigned int maxpages,
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 9ec806f989f2..4b5c7c30cd4d 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1832,6 +1832,8 @@ static ssize_t iov_iter_extract_user_pages(struct iov_iter *i,
gup_flags |= FOLL_WRITE;
if (extraction_flags & ITER_ALLOW_P2PDMA)
gup_flags |= FOLL_PCI_P2PDMA;
+ if (extraction_flags & ITER_ALLOW_LONGTERM)
+ gup_flags |= FOLL_LONGTERM;
if (i->nofault)
gup_flags |= FOLL_NOFAULT;
--
2.25.1
