On Fri, Feb 22, 2013 at 12:56:34PM -0500, Johannes Weiner wrote:
> > > <SNIP>
> > >
> > > : We have a server workload wherein machines with 100G+ of "free" memory
> > > : (used by page cache), scattered but frequent random io reads from 12+
> > > : SSD's, and 5gbps+ of internet traffic, will frequently hit direct reclaim
> > > : in a few different ways.
> > > :
> > > : 1) It'll run into small amounts of reclaim randomly (a few hundred
> > > : thousand).
> > > :
> > > : 2) A burst of reads or traffic can cause extra pressure, which kswapd
> > > : occasionally responds to by freeing up 40g+ of the pagecache all at once
> > > : (!) while pausing the system (Argh).
> > > :
> > > : 3) A blip in an upstream provider or failover from a peer causes the
> > > : kernel to allocate massive amounts of memory for retransmission
> > > : queues/etc, potentially along with buffered IO reads and (some, but not
> > > : often a ton) of new allocations from an application. This paired with 2)
> > > : can cause the box to stall for 15+ seconds.
> > >
> > > Can we prioritise these? 2) looks just awful - kswapd shouldn't just
> > > go off and free 40G of pagecache. Do you know what's actually in that
> > > pagecache? Large number of small files or small number of (very) large
> > > files?
> >
> > We have a handful of huge files (6-12ish 200g+) that are mmap'ed and
> > accessed via address. occasionally madvise (WILLNEED) applied to the
> > address ranges before attempting to use them. There're a mix of other
> > files but nothing significant. The mmap's are READONLY and writes are done
> > via pwrite-ish functions.
> >
> > I could use some guidance on inspecting/tracing the problem. I've been
> > trying to reproduce it in a lab, and respecting to 2)'s issue I've found:
> >
> > - The amount of memory freed back up is either a percentage of total
> > memory or a percentage of free memory. (a machine with 48G of ram will
> > "only" free up an extra 4-7g)
> >
> > - It's most likely to happen after a fresh boot, or if "3 > drop_caches"
> > is applied with the application down. As it fills it seems to get itself
> > into trouble, but becomes more stable after that. Unfortunately 1) and 3)
> > still apply to a stable instance.
> >
> > - Protecting the DMA32 zone with something like "1 1 32" into
> > lowmem_reserve_ratio makes the mass-reclaiming less likely to happen.
> >
> > - While watching "sar -B 1" I'll see kswapd wake up, and scan up to a few
> > hundred thousand pages before finding anything it actually wants to
> > reclaim (low vmeff). I've only been able to reproduce this from a clean
> > start. It can take up to 3 seconds before kswapd starts actually
> > reclaiming pages.
> >
> > - So far as I can tell we're almost exclusively using 0 order allocations.
> > THP is disabled.
> >
> > There's not much dirty memory involved. It's not flushing out writes while
> > reclaiming, it just kills off massive amount of cached memory.
>
> Mapped file pages have to get scanned twice before they are reclaimed
> because we don't have enough usage information after the first scan.
>
> In your case, when you start this workload after a fresh boot or
> dropping the caches, there will be 48G of mapped file pages that have
> never been scanned before and that need to be looked at twice.
>
> Unfortunately, if kswapd does not make progress (and it won't for some
> time at first), it will scan more and more aggressively with
> increasing scan priority. And when the 48G of pages are finally
> cycled, kswapd's scan window is a large percentage of your machine's
> memory, and it will free every single page in it.
>
> I think we should think about capping kswapd zone reclaim cycles just
> as we do for direct reclaim. It's a little ridiculous that it can run
> unbounded and reclaim every page in a zone without ever checking back
> against the watermark. We still increase the scan window evenly when
> we don't make forward progress, but we are more carefully inching zone
> levels back toward the watermarks.
>
While on the surface I think this will appear to work, I worry that it
will cause kswapds priorities to continually reset even when it's under
real pressure as opposed to "failing to reclaim because of use-once".
With nr_to_reclaim always at SWAP_CLUSTER_MAX, we'll hit this check and
reset after each zone scan.
if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
break;
It'll fail the watermark check and restart of course but it does mean we
would call shrink_slab() for every SWAP_CLUSTER_MAX*nr_unbalaced_zones
pages scanned which will have other consequences. It'll behave differently
but not necessarily better.
In general, IO causing anonymous workloads to stall has gotten a lot worse
during the last few kernels without us properly realising it other than
interactivity in the presence of IO has gone down the crapper again. Late
last week I fixed up an mmtests that runs memcachetest as the primary
workload while doing varying amounts of IO in the background and found this
http://www.csn.ul.ie/~mel/postings/reclaim-20130221/global-dhp__parallelio-memcachetest-ext4/hydra/report.html
Snippet looks like this;
3.0.56 3.6.10 3.7.4 3.8.0-rc4
mainline mainline mainline mainline
Ops memcachetest-0M 10125.00 ( 0.00%) 10091.00 ( -0.34%) 11038.00 ( 9.02%) 10864.00 ( 7.30%)
Ops memcachetest-749M 10097.00 ( 0.00%) 8546.00 (-15.36%) 8770.00 (-13.14%) 4872.00 (-51.75%)
Ops memcachetest-1623M 10161.00 ( 0.00%) 3149.00 (-69.01%) 3645.00 (-64.13%) 2760.00 (-72.84%)
Ops memcachetest-2498M 8095.00 ( 0.00%) 2527.00 (-68.78%) 2461.00 (-69.60%) 2282.00 (-71.81%)
Ops memcachetest-3372M 7814.00 ( 0.00%) 2369.00 (-69.68%) 2396.00 (-69.34%) 2323.00 (-70.27%)
Ops memcachetest-4247M 3818.00 ( 0.00%) 2366.00 (-38.03%) 2391.00 (-37.38%) 2274.00 (-40.44%)
Ops memcachetest-5121M 3852.00 ( 0.00%) 2335.00 (-39.38%) 2384.00 (-38.11%) 2233.00 (-42.03%)
This is showing transactions/second -- more the better. 3.0.56 was pretty
bad in itself because a large amount of IO in the background wrecked the
throughput. It's gotten a lot worse since then. 3.8 results have
completed but a quick check tells me the results are no better which is
not surprising as there were no relevant commits since 3.8-rc4.
Ops swapin-0M 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-749M 0.00 ( 0.00%) 36002.00 (-99.00%) 50499.00 (-99.00%) 155135.00 (-99.00%)
Ops swapin-1623M 8.00 ( 0.00%) 176816.00 (-2210100.00%) 172010.00 (-2150025.00%) 206212.00 (-2577550.00%)
Ops swapin-2498M 26291.00 ( 0.00%) 195448.00 (-643.40%) 200911.00 (-664.18%) 209180.00 (-695.63%)
Ops swapin-3372M 27787.00 ( 0.00%) 179221.00 (-544.98%) 183509.00 (-560.41%) 182371.00 (-556.32%)
Ops swapin-4247M 105081.00 ( 0.00%) 157617.00 (-50.00%) 158054.00 (-50.41%) 167478.00 (-59.38%)
Ops swapin-5121M 89589.00 ( 0.00%) 148095.00 (-65.30%) 151012.00 (-68.56%) 159079.00 (-77.57%)
This is indicating that we are making the wrong reclaim decisions
because of the amount of swapins.
Ops majorfaults-0M 1.00 ( 0.00%) 1.00 ( 0.00%) 9.00 (-800.00%) 0.00 ( 0.00%)
Ops majorfaults-749M 2.00 ( 0.00%) 5356.00 (-267700.00%) 7872.00 (-393500.00%) 22472.00 (-1123500.00%)
Ops majorfaults-1623M 30.00 ( 0.00%) 26950.00 (-89733.33%) 25074.00 (-83480.00%) 28815.00 (-95950.00%)
Ops majorfaults-2498M 6459.00 ( 0.00%) 27719.00 (-329.15%) 27904.00 (-332.02%) 29001.00 (-349.00%)
Ops majorfaults-3372M 5133.00 ( 0.00%) 25565.00 (-398.05%) 26444.00 (-415.18%) 25789.00 (-402.42%)
Ops majorfaults-4247M 19822.00 ( 0.00%) 22767.00 (-14.86%) 22936.00 (-15.71%) 23475.00 (-18.43%)
Ops majorfaults-5121M 17689.00 ( 0.00%) 21292.00 (-20.37%) 21820.00 (-23.35%) 22234.00 (-25.69%)
Major faults are also high.
I have not had enough time to investigate this because other bugs cropped
up. I can tell you that it's not bisectable as there are multiple root
causes and it's not always reliably reproducible (with this test at least).
Unfortunately I'm also dropping offline today for a week and then I'll
have to play catchup again when I get back. It's going to be close to 2
weeks before I can start figuring out what went wrong here but I plan to
start with 3.0 and work forward and see how I get on.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>