Interesting! Where abouts in the codebase is the code relating to the username cache? I'd like to take a look at this. Thanks, Nathan. On Wed, Dec 18, 2013 at 5:33 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote: > On 18/12/2013 6:54 p.m., Alex Rousskov wrote: >> On 12/16/2013 10:24 PM, Nathan Hoad wrote: >> >>> While running under this configuration, I've confirmed that memory >>> usage does go up when active, and stays at that level when inactive, >>> allowing some time for timeouts and whatnot. I'm currently switching >>> between the two instances every fifteen minutes. >>> >>> Here is a link to the memory graph for the entire running time of the >>> second process, at 1 minute intervals: >>> http://getoffmalawn.com/static/mem-graph.png. The graph shows memory >>> use steadily increasing during activity, but remaining reasonably >>> stable during inactivity. >> >> I agree that this looks like a memory leak, but (in general) it could >> also be some kind of memory pooling or cache entry accumulation. >> >> >>> Where shall we go from here? >> >> >> I recommend the following next steps: >> >> 1. Set "memory_pools off". >> >> 2. Disable all caching with "cache deny all". >> >> Do you see as similar memory growth pattern after the above two steps? >> >> * If yes: Time for valgrind or ALL,9 debugging. I can help you make that >> choice if needed. You can actually do those things now, without doing >> steps 1 and 2 first, but valgrind and log analysis take time so if we >> can avoid it by eliminating false positives and/or simplifying the >> setup, we should do that first... >> >> * If no: Try re-enabling caching, but using smaller memory [and disk?] >> cache sizes so that a cache gets and stays _full_ way before you run out >> of RAM. This will eliminate cache index growth as a suspect. If your >> disk cache is full already or uses Rock store, then this applies to >> memory cache only. Do you see as similar memory growth pattern after >> re-enabling caching? Are your caches full? > > > The other possibility is username cache in 3.3, since the credentials > used to be tied to the HTTP request details they came from. We had a > memory "leak" bug a while back where the credentials reference to that > request would hold up releasing of the entire transaction state memory. > > Amos
