Minchan Kim <minchan@xxxxxxxxxx> writes:
> On Wed, Jan 05, 2022 at 08:34:40PM -0300, Mauricio Faria de Oliveira wrote:
>> Problem:
>> =======
>>
>> Userspace might read the zero-page instead of actual data from a
>> direct IO read on a block device if the buffers have been called
>> madvise(MADV_FREE) on earlier (this is discussed below) due to a
>> race between page reclaim on MADV_FREE and blkdev direct IO read.
>>
>> Race condition:
>> ==============
>>
>> During page reclaim, the MADV_FREE page check in try_to_unmap_one()
>> checks if the page is not dirty, then discards its PTE (vs remap it
>> back if the page is dirty).
>>
>> However, after try_to_unmap_one() returns to shrink_page_list(), it
>> might keep the page _anyway_ if page_ref_freeze() fails (it expects
>> a single page ref from the isolation).
>>
>> Well, blkdev_direct_IO() gets references for all pages, and on READ
>> operations it sets them dirty later.
>>
>> So, if MADV_FREE pages (i.e., not dirty) are used as buffers (more
>> later) for direct IO read from block devices and page reclaim runs
>> during __blkdev_direct_IO[_simple]() AFTER bio_iov_iter_get_pages()
>> but BEFORE it sets pages dirty, that situation happens.
>>
>> The direct IO read eventually completes. Now, when userspace reads
>> the buffers, the PTE is no longer there and the page fault handler
>> do_anonymous_page() services that with the zero-page, NOT the data!
>>
>> A synthetic reproducer is provided.
>>
>> Page faults:
>> ===========
>>
>> The data read from the block device probably won't generate faults
>> due to DMA (no MMU) but even in the case it wouldn't use DMA, that
>> happens on different virtual addresses (not user-mapped addresses)
>> because `struct bio_vec` stores `struct page` to figure addresses
>> out (which are different from/unrelated to user-mapped addresses)
>> for the data read.
>>
>> Thus userspace reads (to user-mapped addresses) still fault, then
>> do_anonymous_page() gets another `struct page` that would address/
>> map to other memory than the `struct page` used by `struct bio_vec`
>> for the read (which runs correctly as the page wasn't freed due to
>> page_ref_freeze(), and is reclaimed later) -- but even if the page
>> addresses matched, that handler maps the zero-page in the PTE, not
>> that page's memory (on read faults.)
>>
>> If page reclaim happens BEFORE bio_iov_iter_get_pages() the issue
>> doesn't happen, because that faults-in all pages as writeable, so
>> do_anonymous_page() sets up a new page/rmap/PTE, and that is used
>> by direct IO. The userspace reads don't fault as the PTE is there
>> (thus zero-page is not used.)
>>
>> Solution:
>> ========
>>
>> One solution is to check for the expected page reference count in
>> try_to_unmap_one() too, which should be exactly two: one from the
>> isolation (checked by shrink_page_list()), and the other from the
>> rmap (dropped by the discard: label). If that doesn't match, then
>> remap the PTE back, just like page dirty does.
>>
>> The new check in try_to_unmap_one() should be safe in races with
>> bio_iov_iter_get_pages() in get_user_pages() fast and slow paths,
>> as it's done under the PTE lock. The fast path doesn't take that
>> lock but it checks the PTE has changed, then drops the reference
>> and leaves the page for the slow path (which does take that lock).
>>
>> - try_to_unmap_one()
>> - page_vma_mapped_walk()
>> - map_pte() # see pte_offset_map_lock():
>> pte_offset_map()
>> spin_lock()
>> - page_ref_count() # new check
>> - page_vma_mapped_walk_done() # see pte_unmap_unlock():
>> pte_unmap()
>> spin_unlock()
>>
>> - bio_iov_iter_get_pages()
>> - __bio_iov_iter_get_pages()
>> - iov_iter_get_pages()
>> - get_user_pages_fast()
>> - internal_get_user_pages_fast()
>>
>> # fast path
>> - lockless_pages_from_mm()
>> - gup_{pgd,p4d,pud,pmd,pte}_range()
>> ptep = pte_offset_map() # not _lock()
>> pte = ptep_get_lockless(ptep)
>> page = pte_page(pte)
>> try_grab_compound_head(page) # get ref
>> if (pte_val(pte) != pte_val(*ptep))
>> put_compound_head(page) # put ref
>> # leave page for slow path
>> # slow path
>> - __gup_longterm_unlocked()
>> - get_user_pages_unlocked()
>> - __get_user_pages_locked()
>> - __get_user_pages()
>> - follow_{page,p4d,pud,pmd}_mask()
>> - follow_page_pte()
>> ptep = pte_offset_map_lock()
>> pte = *ptep
>> page = vm_normal_page(pte)
>> try_grab_page(page) # get ref
>> pte_unmap_unlock()
>>
>> Regarding transparent hugepages, that number shouldn't change, as
>> MADV_FREE (aka lazyfree) pages are PageAnon() && !PageSwapBacked()
>> (madvise_free_pte_range() -> mark_page_lazyfree() -> lru_lazyfree_fn())
>> thus should reach shrink_page_list() -> split_huge_page_to_list()
>> before try_to_unmap[_one](), so it deals with normal pages only.
>>
>> (And in case unlikely/TTU_SPLIT_HUGE_PMD/split_huge_pmd_address()
>> happens, which it should not or be rare, the page refcount is not
>> two, as the head page counts tail pages, and tail pages have zero.
>> That also prevents checking the head `page` then incorrectly call
>> page_remove_rmap(subpage) for a tail page, that isn't even in the
>> shrink_page_list()'s page_list (an effect of split huge pmd/pmvw),
>> as it might happen today in this unlikely scenario.)
>>
>> MADV_FREE'd buffers:
>> ===================
>>
>> So, back to the "if MADV_FREE pages are used as buffers" note.
>> The case is arguable, and subject to multiple interpretations.
>>
>> The madvise(2) manual page on the MADV_FREE advice value says:
>> - 'After a successful MADV_FREE ... data will be lost when
>> the kernel frees the pages.'
>> - 'the free operation will be canceled if the caller writes
>> into the page' / 'subsequent writes ... will succeed and
>> then [the] kernel cannot free those dirtied pages'
>> - 'If there is no subsequent write, the kernel can free the
>> pages at any time.'
>>
>> Thoughts, questions, considerations...
>> - Since the kernel didn't actually free the page (page_ref_freeze()
>> failed), should the data not have been lost? (on userspace read.)
>> - Should writes performed by the direct IO read be able to cancel
>> the free operation?
>> - Should the direct IO read be considered as 'the caller' too,
>> as it's been requested by 'the caller'?
>> - Should the bio technique to dirty pages on return to userspace
>> (bio_check_pages_dirty() is called/used by __blkdev_direct_IO())
>> be considered in another/special way here?
>> - Should an upcoming write from a previously requested direct IO
>> read be considered as a subsequent write, so the kernel should
>> not free the pages? (as it's known at the time of page reclaim.)
>>
>> Technically, the last point would seem a reasonable consideration
>> and balance, as the madvise(2) manual page apparently (and fairly)
>> seem to assume that 'writes' are memory access from the userspace
>> process (not explicitly considering writes from the kernel or its
>> corner cases; again, fairly).. plus the kernel fix implementation
>> for the corner case of the largely 'non-atomic write' encompassed
>> by a direct IO read operation, is relatively simple; and it helps.
>>
>> Reproducer:
>> ==========
>>
>> @ test.c (simplified, but works)
>>
>> #define _GNU_SOURCE
>> #include <fcntl.h>
>> #include <stdio.h>
>> #include <unistd.h>
>> #include <sys/mman.h>
>>
>> int main() {
>> int fd, i;
>> char *buf;
>>
>> fd = open(DEV, O_RDONLY | O_DIRECT);
>>
>> buf = mmap(NULL, BUF_SIZE, PROT_READ | PROT_WRITE,
>> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>
>> for (i = 0; i < BUF_SIZE; i += PAGE_SIZE)
>> buf[i] = 1; // init to non-zero
>>
>> madvise(buf, BUF_SIZE, MADV_FREE);
>>
>> read(fd, buf, BUF_SIZE);
>>
>> for (i = 0; i < BUF_SIZE; i += PAGE_SIZE)
>> printf("%p: 0x%x\n", &buf[i], buf[i]);
>>
>> return 0;
>> }
>>
>> @ block/fops.c (formerly fs/block_dev.c)
>>
>> +#include <linux/swap.h>
>> ...
>> ... __blkdev_direct_IO[_simple](...)
>> {
>> ...
>> + if (!strcmp(current->comm, "good"))
>> + shrink_all_memory(ULONG_MAX);
>> +
>> ret = bio_iov_iter_get_pages(...);
>> +
>> + if (!strcmp(current->comm, "bad"))
>> + shrink_all_memory(ULONG_MAX);
>> ...
>> }
>>
>> @ shell
>>
>> # yes | dd of=test.img bs=1k count=16
>> # DEV=$(losetup -f --show test.img)
>> # gcc -DDEV=\"$DEV\" -DBUF_SIZE=16384 -DPAGE_SIZE=4096 test.c -o test
>>
>> # od -tx1 $DEV
>> 0000000 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a
>> *
>> 0040000
>>
>> # mv test good
>> # ./good
>> 0x7f1509206000: 0x79
>> 0x7f1509207000: 0x79
>> 0x7f1509208000: 0x79
>> 0x7f1509209000: 0x79
>>
>> # mv good bad
>> # ./bad
>> 0x7fd87272f000: 0x0
>> 0x7fd872730000: 0x0
>> 0x7fd872731000: 0x0
>> 0x7fd872732000: 0x0
>>
>> Ceph/TCMalloc:
>> =============
>>
>> For documentation purposes, the use case driving the analysis/fix
>> is Ceph on Ubuntu 18.04, as the TCMalloc library there still uses
>> MADV_FREE to release unused memory to the system from the mmap'ed
>> page heap (might be committed back/used again; it's not munmap'ed.)
>> - PageHeap::DecommitSpan() -> TCMalloc_SystemRelease() -> madvise()
>> - PageHeap::CommitSpan() -> TCMalloc_SystemCommit() -> do nothing.
>>
>> Note: TCMalloc switched back to MADV_DONTNEED a few commits after
>> the release in Ubuntu 18.04 (google-perftools/gperftools 2.5), so
>> the issue just 'disappeared' on Ceph on later Ubuntu releases but
>> is still present in the kernel, and can be hit by other use cases.
>>
>> The observed issue seems to be the old Ceph bug #22464 [1], where
>> checksum mismatches are observed (and instrumentation with buffer
>> dumps shows zero-pages read from mmap'ed/MADV_FREE'd page ranges).
>>
>> The issue in Ceph was reasonably deemed a kernel bug (comment #50)
>> and mostly worked around with a retry mechanism, but other parts
>> of Ceph could still hit that (rocksdb). Anyway, it's less likely
>> to be hit again as TCMalloc switched out of MADV_FREE by default.
>>
>> (Some kernel versions/reports from the Ceph bug, and relation with
>> the MADV_FREE introduction/changes; TCMalloc versions not checked.)
>> - 4.4 good
>> - 4.5 (madv_free: introduction)
>> - 4.9 bad
>> - 4.10 good? maybe a swapless system
>> - 4.12 (madv_free: no longer free instantly on swapless systems)
>> - 4.13 bad
>>
>> [1] https://tracker.ceph.com/issues/22464
>>
>> Thanks:
>> ======
>>
>> Several people contributed to analysis/discussions/tests/reproducers
>> in the first stages when drilling down on ceph/tcmalloc/linux kernel:
>>
>> - Dan Hill <daniel.hill@xxxxxxxxxxxxx>
>> - Dan Streetman <dan.streetman@xxxxxxxxxxxxx>
>> - Dongdong Tao <dongdong.tao@xxxxxxxxxxxxx>
>> - Gavin Guo <gavin.guo@xxxxxxxxxxxxx>
>> - Gerald Yang <gerald.yang@xxxxxxxxxxxxx>
>> - Heitor Alves de Siqueira <halves@xxxxxxxxxxxxx>
>> - Ioanna Alifieraki <ioanna-maria.alifieraki@xxxxxxxxxxxxx>
>> - Jay Vosburgh <jay.vosburgh@xxxxxxxxxxxxx>
>> - Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx>
>> - Ponnuvel Palaniyappan <ponnuvel.palaniyappan@xxxxxxxxxxxxx>
>>
>> v2: check refcount against mapcount rather than a static 2.
>> Thanks: Minchan Kim <minchan@xxxxxxxxxx>
>>
>> Signed-off-by: Mauricio Faria de Oliveira <mfo@xxxxxxxxxxxxx>
>> ---
>> mm/rmap.c | 15 ++++++++++++++-
>> mm/vmscan.c | 2 +-
>> 2 files changed, 15 insertions(+), 2 deletions(-)
>>
>> diff --git a/mm/rmap.c b/mm/rmap.c
>> index 163ac4e6bcee..8671de473c25 100644
>> --- a/mm/rmap.c
>> +++ b/mm/rmap.c
>> @@ -1570,7 +1570,20 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
>>
>> /* MADV_FREE page check */
>> if (!PageSwapBacked(page)) {
>> - if (!PageDirty(page)) {
>> + int ref_count = page_ref_count(page);
>> + int map_count = page_mapcount(page);
>> +
>> + /*
>> + * The only page refs must be from the isolation
>> + * (checked by the caller shrink_page_list() too)
>> + * and one or more rmap's (dropped by discard:).
>> + *
>> + * Check the reference count before dirty flag
>> + * with memory barrier; see __remove_mapping().
>> + */
>> + smp_rmb();
>> + if ((ref_count - 1 == map_count) &&
>> + !PageDirty(page)) {
>
> Thanks. Looks good to me.
>
> I'd like to invite more folks for better eyes since this could
> be a rather subtle issue.
Looks good to me too! Please feel free to add my
Reviewed-by: "Huang, Ying" <ying.huang@xxxxxxxxx>
Best Regards,
Huang, Ying
