On Fri, Sep 3, 2010 at 2:06 PM, Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> wrote:
We don't have any quantitative data on the effect of these excess tlbOn Mon, 30 Aug 2010 00:43:48 +0900
Minchan Kim <minchan.kim@xxxxxxxxx> wrote:
> Ying Han reported that backing aging of anon pages in no swap system
> causes unnecessary TLB flush.
>
> When I sent a patch(69c8548175), I wanted this patch but Rik pointed out
> and allowed aging of anon pages to give a chance to promote from inactive
> to active LRU.
>
> It has a two problem.
>
> 1) non-swap system
>
> Never make sense to age anon pages.
>
> 2) swap configured but still doesn't swapon
>
> It doesn't make sense to age anon pages until swap-on time.
> But it's arguable. If we have aged anon pages by swapon, VM have moved
> anon pages from active to inactive. And in the time swapon by admin,
> the VM can't reclaim hot pages so we can protect hot pages swapout.
>
> But let's think about it. When does swap-on happen? It depends on admin.
> we can't expect it. Nonetheless, we have done aging of anon pages to
> protect hot pages swapout. It means we lost run time overhead when
> below high watermark but gain hot page swap-[in/out] overhead when VM
> decide swapout. Is it true? Let's think more detail.
> We don't promote anon pages in case of non-swap system. So even though
> VM does aging of anon pages, the pages would be in inactive LRU for a long
> time. It means many of pages in there would mark access bit again. So access
> bit hot/code separation would be pointless.
>
> This patch prevents unnecessary anon pages demotion in not-swapon and
> non-configured swap system. Of course, it could make side effect that
> hot anon pages could swap out when admin does swap on.
> But I think sooner or later it would be steady state.
> So it's not a big problem.
> We could lose someting but gain more thing(TLB flush and unnecessary
> function call to demote anon pages).
>
> I used total_swap_pages because we want to age anon pages
> even though swap full happens.
flushes, which makes it difficult to decide which kernel versions
should receive this patch.
Help?
Andrew:
We observed the degradation on 2.6.34 compared to 2.6.26 kernel. The workload we are running is doing 4k-random-write which runs about 3-4 minutes. We captured the TLB shootsdowns before/after:
Before the change:
TLB: 29435 22208 37146 25332 47952 43698 43545 40297 49043 44843 46127 50959 47592 46233 43698 44690 TLB shootdowns [HSUM = 662798 ]
After the change:
TLB: 2340 3113 1547 1472 2944 4194 2181 1212 2607 4373 1690 1446 2310 3784 1744 1134 TLB shootdowns [HSUM = 38091 ]
Also worthy to mention, we are running in fake numa system where each fake node is 128M size. That makes differences on the check inactive_anon_is_low() since the active/inactive ratio falls to 1.
--Ying