Dear Michal, Am 11.09.19 um 08:12 schrieb Stefan Priebe - Profihost AG: > Hi Michal, > Am 10.09.19 um 15:24 schrieb Michal Hocko: >> On Tue 10-09-19 15:14:45, Stefan Priebe - Profihost AG wrote: >>> Am 10.09.19 um 15:05 schrieb Stefan Priebe - Profihost AG: >>>> >>>> Am 10.09.19 um 14:57 schrieb Michal Hocko: >>>>> On Tue 10-09-19 14:45:37, Stefan Priebe - Profihost AG wrote: >>>>>> Hello Michal, >>>>>> >>>>>> ok this might take a long time. Attached you'll find a graph from a >>>>>> fresh boot what happens over time (here 17 August to 30 August). Memory >>>>>> Usage decreases as well as cache but slowly and only over time and days. >>>>>> >>>>>> So it might take 2-3 weeks running Kernel 5.3 to see what happens. >>>>> >>>>> No problem. Just make sure to collect the requested data from the time >>>>> you see the actual problem. Btw. you try my very dumb scriplets to get >>>>> an idea of how much memory gets reclaimed due to THP. >>>> >>>> You mean your sed and sort on top of the trace file? No i did not with >>>> the current 5.3 kernel do you think it will show anything interesting? >>>> Which line shows me how much memory gets reclaimed due to THP? >> >> Please re-read http://lkml.kernel.org/r/20190910082919.GL2063@xxxxxxxxxxxxxx >> Each command has a commented output. If you see nunmber of reclaimed >> pages to be large for GFP_TRANSHUGE then you are seeing a similar >> problem. >> >>> Is something like a kernel memory leak possible? Or wouldn't this end up >>> in having a lot of free memory which doesn't seem usable. >> >> I would be really surprised if this was the case. >> >>> I also wonder why a reclaim takes place when there is enough memory. >> >> This is not clear yet and it might be a bug that has been fixed since >> 4.18. That's why we need to see whether the same is pattern is happening >> with 5.3 as well. Kernel 5.2.14 is now running since exactly 7 days and now we can easaly view a trend i', not sure if i should post graphs. Cache size is continuously shrinking while memfree is rising. While there were 4,5GB free in avg in the beginnen we now have an avg of 8GB free memory. Cache has shrinked from avg 24G to avg 18G. Memory pressure has rised from avg 0% to avg 0.1% - not much but if you look at the graphs it's continuously rising while cache is shrinking and memfree is rising. Which values should i collect now? Greets, Stefan