On Fri, Feb 10, 2017 at 3:01 AM, Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx> wrote: > On Thu, Feb 09, 2017 at 06:46:57PM -0800, Andy Lutomirski wrote: >> > try_to_unmap_flush then flushes the entire TLB as the cost of targetted >> > a specific page to flush was so high (both maintaining the PFNs and the >> > individual flush operations). >> >> I could just maybe make it possible to remotely poke a CPU to record >> which mms need flushing, but the possible races there are a bit >> terrifying. >> > > The overhead is concerning. You may incur a remote cache miss accessing the > data which is costly or you have to send an IPI which is also severe. You > could attempt to do the same as the scheduler and directly modify if the > CPUs share cache and IPI otherwise but you're looking at a lot of overhead > either way. I think all of these approaches suck and I'll give up on this particular avenue. >> Anyway, I need to rework the arch code to make this work at all. >> Currently I'm taking a spinlock per mm when flushing that mm, but that >> would mean I need to *sort* the list to flush more than one at a time, >> and that just sounds nasty. I can probably get rid of the spinlock. >> > > That all sounds fairly nasty. Don't get me wrong, I think you can make > it functionally work but it's a severe uphill battle. > > The key concern that it'll be evaluated against is that any complexity has > to be less than doing a "batched full TLB flush and refill". The refill is > expected to be cheap as the page table structures are likely to be cache hot. > It was way cheaper than trying to be clever about flushing individual TLB > entries. You're assuming that Intel CPUs are more sensible than they really are. My suspicion, based on some benchmarking, is that "switch pgd and clear the TLB" is very slow because of the "clear the TLB bit" -- that is, above and beyond the refill cost, merely clearing the TLB takes hundreds of cycles. So I'd like to minimize unnecessary flushes. Unfortunately, "flush the TLB for all ASIDs" and "flush everything" are also both very very slow. Sigh. > I recognise that you'll be trying to balance this against processes > that are carefully isolated that do not want interference from unrelated > processes doing a TLB flush but it'll be hard to prove that it's worth it. > > It's almost certain that this will be Linus' primary concern > given his contributions to similar conversations in the past > (e.g. https://lkml.org/lkml/2015/6/25/666). It's also likely to be of > major concern to Ingo (e.g. https://lkml.org/lkml/2015/6/9/276) as he had > valid objections against clever flushing at the time the batching was > introduced. Based on previous experience, I have my own concerns but I > don't count as I'm highlighing them now :P I fully agree with those objections, but back then we didn't have the capability to avoid a flush when switching mms. All that being said, I agree: making this stuff too complicated is a bad idea. > > The outcome of the TLB batch flushiing discussion was that it was way > cheaper to flush the full TLB and take the refill cost than flushing > individual pages which had the cost of tracking the PFNs and the cost of > each individual page flush operation. > > The current code is basically "build a cpumask and flush the TLB for > multiple entries". We're talking about complex tracking of mm's with > difficult locking, potential remote cache misses, potentially more IPIs or > alternatively doing allocations from reclaim context. It'll be difficult > to prove that doing this in the name of flushing ASID is cheaper and > universally a good idea than just flushing the entire TLB. > Maybe there's a middle ground. I could keep track of whether more than one mm is targetted in a deferred flush and just flush everything if so. As a future improvement, I or someone else could add: struct mm_struct *mms[16]; int num_mms; to struct tlbflush_unmap_batch. if num_mms > 16, then this just means that we've given up on tracking them all and we do the global flush, and, if not, we could teach the IPI handler to understand a list of target mms. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
