On 2022/8/19 11:19, Muchun Song wrote: > > >> On Aug 18, 2022, at 20:58, Miaohe Lin <linmiaohe@xxxxxxxxxx> wrote: >> >> On 2022/8/18 17:18, Muchun Song wrote: >>> >>> >>>> On Aug 18, 2022, at 16:54, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote: >>>> >>>> >>>> >>>> On 8/18/2022 4:40 PM, Muchun Song wrote: >>>>> >>>>> >>>>>> On Aug 18, 2022, at 16:32, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 8/18/2022 3:59 PM, Muchun Song wrote: >>>>>>> >>>>>>> >>>>>>>> On Aug 18, 2022, at 15:52, Miaohe Lin <linmiaohe@xxxxxxxxxx> wrote: >>>>>>>> >>>>>>>> On 2022/8/18 10:47, Muchun Song wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Aug 18, 2022, at 10:00, Yin, Fengwei <fengwei.yin@xxxxxxxxx> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 8/18/2022 9:55 AM, Miaohe Lin wrote: >>>>>>>>>>>>>> /* >>>>>>>>>>>>>> * The memory barrier inside __SetPageUptodate makes sure that >>>>>>>>>>>>>> * preceding stores to the page contents become visible before >>>>>>>>>>>>>> * the set_pte_at() write. >>>>>>>>>>>>>> */ >>>>>>>>>>>>>> __SetPageUptodate(page); >>>>>>>>>>>>> IIUC, the case here we should make sure others (CPUs) can see new page’s >>>>>>>>>>>>> contents after they have saw PG_uptodate is set. I think commit 0ed361dec369 >>>>>>>>>>>>> can tell us more details. >>>>>>>>>>>>> >>>>>>>>>>>>> I also looked at commit 52f37629fd3c to see why we need a barrier before >>>>>>>>>>>>> set_pte_at(), but I didn’t find any info to explain why. I guess we want >>>>>>>>>>>>> to make sure the order between the page’s contents and subsequent memory >>>>>>>>>>>>> accesses using the corresponding virtual address, do you agree with this? >>>>>>>>>>>> This is my understanding also. Thanks. >>>>>>>>>>> That's also my understanding. Thanks both. >>>>>>>>>> I have an unclear thing (not related with this patch directly): Who is response >>>>>>>>>> for the read barrier in the read side in this case? >>>>>>>>>> >>>>>>>>>> For SetPageUptodate, there are paring write/read memory barrier. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I have the same question. So I think the example proposed by Miaohe is a little >>>>>>>>> difference from the case (hugetlb_vmemmap) here. >>>>>>>> >>>>>>>> Per my understanding, memory barrier in PageUptodate() is needed because user might access the >>>>>>>> page contents using page_address() (corresponding pagetable entry already exists) soon. But for >>>>>>>> the above proposed case, if user wants to access the page contents, the corresponding pagetable >>>>>>>> should be visible first or the page contents can't be accessed. So there should be a data dependency >>>>>>>> acting as memory barrier between pagetable entry is loaded and page contents is accessed. >>>>>>>> Or am I miss something? >>>>>>> >>>>>>> Yep, it is a data dependency. The difference between hugetlb_vmemmap and PageUptodate() is that >>>>>>> the page table (a pointer to the mapped page frame) is loaded by MMU while PageUptodate() is >>>>>>> loaded by CPU. Seems like the data dependency should be inserted between the MMU access and the CPU >>>>>>> access. Maybe it is hardware’s guarantee? >>>>>> I just found the comment in pmd_install() explained why most arch has no read >>>>> >>>>> I think pmd_install() is a little different as well. We should make sure the >>>>> page table walker (like GUP) see the correct PTE entry after they see the pmd >>>>> entry. >>>> >>>> The difference I can see is that pmd/pte thing has both hardware page walker and >>>> software page walker (like GUP) as read side. While the case here only has hardware >>>> page walker as read side. But I suppose the memory barrier requirement still apply >>>> here. >>> >>> I am not against this change. Just in order to make me get a better understanding of >>> hardware behavior. >>> >>>> >>>> Maybe we could do a test: add large delay between reset_struct_page() and set_pte_at? >>> >>> Hi Miaohe, >>> >>> Would you mind doing this test? One thread do vmemmap_restore_pte(), another thread >>> detect if it can see a tail page with PG_head after the previous thread has executed >>> set_pte_at(). >> >> Will it be easier to construct the memory reorder manually like below? >> >> vmemmap_restore_pte() >> ... >> set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); >> /* might a delay. */ >> copy_page(to, (void *)walk->reuse_addr); >> reset_struct_pages(to); > > > Well, you have changed the code ordering. I thought we don’t change the code > ordering. Just let the hardware do reordering. The ideal scenario would be > as follows. > > > CPU0: CPU1: > > vmemmap_restore_pte() > copy_page(to, (void *)walk->reuse_addr); > reset_struct_pages(to); // clear the tail page’s PG_head > set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); > // Detect if it can see a tail page with PG_head. > > I should admit it is a little difficult to construct the scenario. After more > thought, I think here should be inserted a barrier. So: > > Reviewed-by: Muchun Song <songmuchun@xxxxxxxxxxxxx> Many thanks both for review and discussion. :) Thanks, Miaohe Lin
