On 07/17/2014 02:49 AM, Martin Sperl wrote: > This is why I have been mentioning all the pools that show similar (wavy) > memory increase pattern. There must be one of those that is the root of > all the others. Unfortunately, I do not share your certainty. As I see it, at least two theories more-or-less fit your data: A) MemObjects are leaking. B) Your memory cache is growing, explaining some or all of the pools growth and muddying the waters along the way. Something unrelated to MemObjects is leaking. That something may or may not be pooled. My bet is on (B). If you do not mind reminding me, does accounted-for memory pools growth correspond to the actual/total Squid memory growth? Or is the "pooled" vs "total/real" gap widening with time? > the strange thing is that if you look at the > distribution of vm_objects, then you see that they have expired long ago > (16267 days ago to be exact, so with EX:-1 - 42511 exactly). > If these have been expired, then why would they get loaded into memory? because you have free cache space and Squid can still serve that cached data (after revalidation)? In general, there is no benefit from purging something from a non-full cache, especially if that something is reusable. I have not checked the code and responses producing those "16267 days" stats so I treat that number with a grain of salt. :-) > Well: SUM(inmem_hi) is memory for payload (possibly without headers) against > which we compare the cache_mem against. > > But if the squid process consumes 9GB, then there must be more than a factor > of 2 of overhead so that we get to those 9GB. Yes if (A) above is correct. If (B) above is correct, that "overhead" is a leak of something other than memory cache entries. > Report-file(with Date+Time) psmemory MemObjects kb/MemObject > report-20140708-234225.txt 1795932 123979 6.03 ... > report-20140717-000001.txt 10662148 845404 11.37 > Which shows that the average size/memObject is increasing constantly! Yes, and that is one of the reasons I bet on (B). (B) is a lot less surprising than a constantly growing MemObject overhead that (A) has to explain :-). Running with a small cache size (i.e., with a full cache) would confirm that. > Another point here more on the reporting/debugging side: > Total space in arena: -2034044 KB Yes, this is a known problem mostly outside of Squid control: http://www.squid-cache.org/mail-archive/squid-users/201402/0296.html Squid should probably stop using that API [when it can detect that it is broken]. And yes, I know that my response does not really help you. I just wanted to make sure you consider both (A) and (B) theories in your investigation. Alex. P.S. Another suggestion that you probably cannot use in your environment is to run trunk or a trunk-based branch. Perhaps the leak you are after has been fixed. And if not, somebody may be more motivated to help you find it in trunk.
