Re: PROBLEM: Memory leaking when running kubernetes cronjobs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Daniel!

On Tue, Sep 25, 2018 at 09:15:20AM +0000, Daniel McGinnes wrote:
> Hi Roman,
> 
> I left it running with the patches for a longer period (with no memory 
> pressure), and it leaked at very similar rate to without the patches.
> 
> I then started applying some memory pressure (Ran stress --vm 16 
> --vm-bytes 1772864000 -t 300 for 5 minutes, then sleep for 5 mins in a 
> continuous loop).
> 
> When I started running stress the MemAvailable increased by ~2GB - but 
> this still left ~4GB "leaked". The interesting thing is, I left the test 
> running whilst the memory pressure loop was running for ~ 1 day so far, 
> and it looks like no additional memory has been "leaked". I have tried 
> drop_caches and it doesn't clear any additional memory. So it looks to me 
> like if memory pressure is applied whilst the test is running it doesn't 
> leak additional memory - but it won't free up all memory that was 
> previously leaked.. How does this match with 
> your expected behaviour?

Yeah, this is pretty match what I was expecting.

I'd bet that if you'll look at the number of dying cgroups, you'll find that my
patches caused them to be reclaimed once the memory pressure was applied (either
after or during the test).

But in the case, when you apply the memory pressure after, the fragmentation in
the per-cpu memory allocator is the reason why you are not getting the memory
back. It's not a real "leak", the memory can be perfectly used for per-cpu
allocations (had you create a big number of cgroups again, for example).

If you have per-cpu stats available on your host (/sys/kernel/debug/percpu_stats),
please, post it, as it can confirm or refute my theory.

Thanks!



[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]     [Monitors]

  Powered by Linux