Hi Alex! Well as for data: I have posted the raw data as an excel sheet including graphs to the ticket: http://bugs.squid-cache.org/show_bug.cgi?id=4084 Especially the tab: "ChartKBto100%" show the wavy pattern of the memory pools (graphed as % of last measured value) And then you have the same, but graphing the delta to the previous hour: "ChartKBto100%Differencial" And finally there is "ChartInternal Data Structures" which graphs the behavior of the 4 different basic structures. Those wavy pattern I have mentioned are visible in each of those graphs and follow our traffic curve. Well - as for the theories: If it is "underestimating the real memory-footprint of a mem_object", then: a reduction of the memory cache size to 2GB should show a leveling of If it is instead wasted memory per request, then decreasing the memory settings would not have an impact. Here some more stats on Requests distribution with regards to cache hit/miss and status codes for a 20hour period: Count CacheStatus-StatusCode 11846509 total requests 6425676 TCP_MEM_HIT-200 1462630 TCP_MISS-200 1460093 TAG_NONE-302 1378134 TCP_IMS_HIT-304 390021 TCP_MISS-302 244197 TCP_REFRESH_MODIFIED-200 101942 TCP_NEGATIVE_HIT-404 88726 TAG_NONE-400 69045 TCP_REFRESH_UNMODIFIED-200 33006 TCP_MISS_ABORTED-0 30973 TAG_NONE_ABORTED-0 26063 TCP_MISS-304 23619 TCP_MISS-301 20107 TCP_DENIED_REPLY-302 17303 TCP_MISS-404 17267 TCP_MISS_ABORTED-200 16727 TCP_DENIED-302 8215 TCP_MEM_HIT-404 7921 TCP_HIT-200 7478 TCP_MEM_HIT_ABORTED-200 4909 TCP_MISS-303 3914 TCP_MISS-206 2746 TCP_DENIED-403 1448 TCP_MEM_HIT-206 777 TAG_NONE-405 670 TCP_MISS_TIMEDOUT-200 448 TCP_MISS_ABORTED-206 390 TCP_MISS-500 310 TCP_HIT_ABORTED-0 240 TCP_DENIED_REPLY-307 232 TCP_MEM_HIT_TIMEDOUT-200 169 TCP_MISS-502 146 TCP_MISS-503 139 TCP_DENIED_ABORTED-302 132 TCP_REFRESH_MODIFIED-302 69 TCP_NEGATIVE_HIT-500 62 TCP_REFRESH_UNMODIFIED-206 60 TCP_NEGATIVE_HIT-502 54 TCP_REFRESH_MODIFIED-206 50 TCP_REFRESH_UNMODIFIED_ABORTED-200 44 TCP_MISS-403 44 TAG_NONE_ABORTED-400 42 TCP_REFRESH_FAIL_ERR-500 36 TCP_REFRESH_MODIFIED_ABORTED-200 35 TCP_DENIED_ABORTED-403 33 TCP_NEGATIVE_HIT-403 30 TCP_DENIED-307 21 TCP_CLIENT_REFRESH_MISS-200 19 TCP_MISS_ABORTED-302 16 TCP_MISS-400 8 TCP_MISS-504 8 TCP_IMS_HIT_ABORTED-304 8 TCP_HIT_ABORTED-200 5 TCP_MISS-416 5 TCP_MEM_HIT_ABORTED-206 4 TCP_REFRESH_MODIFIED-404 4 TCP_REFRESH_FAIL_OLD-200 4 TCP_NEGATIVE_HIT_ABORTED-404 4 TCP_NEGATIVE_HIT-503 4 TCP_MISS_TIMEDOUT-206 4 TCP_MISS-204 4 TCP_MEM_HIT-500 4 TAG_NONE_TIMEDOUT-400 3 TCP_NEGATIVE_HIT-400 2 TCP_REFRESH_UNMODIFIED_TIMEDOUT-200 2 TCP_HIT_TIMEDOUT-200 1 TCP_SWAPFAIL_MISS-200 1 TCP_REFRESH_MODIFIED_TIMEDOUT-200 1 TCP_MISS_ABORTED-404 1 TCP_DENIED_REPLY_ABORTED-302 I will try to do some more analysis including latest data and share the result as well - this time also correlating TPS... Martin > -----Original Message----- > From: Alex Rousskov [mailto:rousskov@xxxxxxxxxxxxxxxxxxxxxxx] > Sent: Donnerstag, 17. Juli 2014 23:03 > To: squid-users@xxxxxxxxxxxxxxx > Cc: Martin Sperl > Subject: Re: squid: Memory utilization higher than expected > since moving from 3.3 to 3.4 and Vary: working > > On 07/17/2014 02:49 AM, Martin Sperl wrote: > > This is why I have been mentioning all the pools that show similar (wavy) > > memory increase pattern. There must be one of those that is the root of > > all the others. > > Unfortunately, I do not share your certainty. As I see it, at least two > theories more-or-less fit your data: > > A) MemObjects are leaking. > > B) Your memory cache is growing, explaining some or all of the pools > growth and muddying the waters along the way. Something unrelated to > MemObjects is leaking. That something may or may not be pooled. > > My bet is on (B). > > If you do not mind reminding me, does accounted-for memory pools growth > correspond to the actual/total Squid memory growth? Or is the "pooled" > vs "total/real" gap widening with time? > > > > the strange thing is that if you look at the > > distribution of vm_objects, then you see that they have expired long ago > > (16267 days ago to be exact, so with EX:-1 - 42511 exactly). > > If these have been expired, then why would they get loaded into memory? > > because you have free cache space and Squid can still serve that cached > data (after revalidation)? In general, there is no benefit from purging > something from a non-full cache, especially if that something is reusable. > > I have not checked the code and responses producing those "16267 days" > stats so I treat that number with a grain of salt. :-) > > > > Well: SUM(inmem_hi) is memory for payload (possibly without headers) > against > > which we compare the cache_mem against. > > > > But if the squid process consumes 9GB, then there must be more than a > factor > > of 2 of overhead so that we get to those 9GB. > > Yes if (A) above is correct. If (B) above is correct, that "overhead" is > a leak of something other than memory cache entries. > > > > Report-file(with Date+Time) psmemory MemObjects kb/MemObject > > report-20140708-234225.txt 1795932 123979 6.03 > ... > > report-20140717-000001.txt 10662148 845404 11.37 > > > Which shows that the average size/memObject is increasing constantly! > > Yes, and that is one of the reasons I bet on (B). (B) is a lot less > surprising than a constantly growing MemObject overhead that (A) has to > explain :-). Running with a small cache size (i.e., with a full cache) > would confirm that. > > > > > Another point here more on the reporting/debugging side: > > Total space in arena: -2034044 KB > > Yes, this is a known problem mostly outside of Squid control: > http://www.squid-cache.org/mail-archive/squid-users/201402/0296.html > > Squid should probably stop using that API [when it can detect that it is > broken]. > > > And yes, I know that my response does not really help you. I just wanted > to make sure you consider both (A) and (B) theories in your investigation. > > > Alex. > P.S. Another suggestion that you probably cannot use in your environment > is to run trunk or a trunk-based branch. Perhaps the leak you are after > has been fixed. And if not, somebody may be more motivated to help you > find it in trunk. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp
