Hi Roman, I have original memleak.py working, let me know when you have the modified version ready. Thanks again for all the help with this! thanks, Dan McGinnes IBM Cloud - Containers performance Int Tel: 247359 Ext Tel: 01962 817359 Notes: Daniel McGinnes/UK/IBM Email: MCGINNES@xxxxxxxxxx IBM (UK) Ltd, Hursley Park,Winchester,Hampshire, SO21 2JN From: Roman Gushchin <guro@xxxxxx> To: Daniel McGinnes <MCGINNES@xxxxxxxxxx> Cc: "cgroups@xxxxxxxxxxxxxxx" <cgroups@xxxxxxxxxxxxxxx>, Nathaniel Rockwell <nrockwell@xxxxxxxxxx> Date: 26/09/2018 14:06 Subject: Re: PROBLEM: Memory leaking when running kubernetes cronjobs On Wed, Sep 26, 2018 at 12:52:22PM +0000, Daniel McGinnes wrote: > Hi, > > I plotted the charts with Percpu & SUnreclaim from /proc/meminfo. Very > interesting that it looks like Percpu increases during the test at a > similar rate MemAvailable is dropping (I plotted them on different axis so > it was easier to see). It is also noticeable that when MemAvailable > plateaus so does Percpu. > > > > So looks like sometimes Percpu increases even though nr_dying_descendant > reamins stable (I had to remove from the chart otherwise there was too > much data - but it has remained stable at 400-500 since 6 hours) - > although it does look like eventually Percpu & MemAvailable have > stabilised (although need to monitor for longer to be sure - given it > stabilised for a few hours 4 hours in, and then started increasing again) Hm, interesting. Is it current version from the Linus's tree? Anyway, let's try to figure out what's taking per-cpu memory. I've used a modified version of memleak.py from bcc ( https://github.com/iovisor/bcc ) to debug such things. Please, make the original memleak.py work for you, and I'll try to find a modified version for tracing the per-cpu allocator. Thanks! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
