Hi Marcus, The 5GB is on a squid on a different server... one not running under valgrind. I've restarted that process since my last e-mail so it's using much less mem now, but be assured it will eventually creep back up to 5GB and beyond. I get the 5GB number from top and ps... they show roughly that number for both virtual and resident. It's a real problem... the machines will start swapping like crazy when they exhaust physical RAM and the servers slow down badly. Left for long enough, they'll eventually crash with a "malloc() returned null" exception or something along those lines. It's a real leak. Amos identified some leaks from my valgrind output and when those make it into a daily build I'll try again and report my findings. Thanks -Ty On Sep 27, 2012, at 5:57 PM, Marcus Kool <marcus.kool@xxxxxxxxxxxxxxx> wrote: > > > On 09/27/2012 03:39 PM, tcr@xxxxxxxxxxxx wrote: >> Some new stuff has been logged by valgrind overnight- it's at the bottom of this e-mail. Looks like mostly dupes but I see some that might be new. >> >> However I still don't see any accounting for the amount of leakage I'm seeing. Right now, squid on one of my servers is using a whopping 5GB of memory, yet when I do mgr:mem on it, it only tells me that the total of all allocations is about 600 MB. > > How do you get the information of "5 GB" ? > > Note that Valgrind uses various techniques for detecting programming errors and > one of them is to not actually release memory when a program want to release it, > but instead administers it as unreleased to detect bugs like > "accessing released memory". See the Valgrind documentation for more information. > >> How can I find out what all those allocations are coming from, since mgr:mem doesn't seem to have any accounting of them? >> >> Thank you >> -Ty >> >> >> >> ==26647== Invalid read of size 8 >> ==26647== at 0x6CE3D44: __strspn_sse42 (in /lib64/libc-2.12.so) >> ==26647== by 0x535F79: strListGetItem (HttpHeaderTools.cc:259) >> ==26647== by 0x51FA07: httpMakeVaryMark (http.cc:538) >> ==26647== by 0x4BB11A: varyEvaluateMatch (client_side.cc:3812) >> ==26647== by 0x4D44B2: clientReplyContext::cacheHit(StoreIOBuffer) (client_side_reply.cc:483) >> ==26647== by 0x57743C: store_client::callback(long, bool) (store_client.cc:165) >> ==26647== by 0x5778A4: store_client::scheduleMemRead() (store_client.cc:454) >> ==26647== by 0x577CEA: store_client::doCopy(StoreEntry*) (store_client.cc:383) >> ==26647== by 0x577FA5: storeClientCopy2(StoreEntry*, store_client*) (store_client.cc:337) >> ==26647== by 0x4D323B: clientReplyContext::doGetMoreData() (client_side_reply.cc:1751) >> ==26647== by 0x4D5614: ClientHttpRequest::httpStart() (client_side_request.cc:1358) >> ==26647== by 0x4D770A: ClientHttpRequest::doCallouts() (client_side_request.cc:1604) >> ==26647== Address 0x1cb6eba8 is 0 bytes after a block of size 40 alloc'd >> ==26647== at 0x4C25A28: calloc (vg_replace_malloc.c:467) >> ==26647== by 0x65B441: xcalloc (xalloc.cc:75) >> ==26647== by 0x657B99: MemPoolMalloc::allocate() (MemPoolMalloc.cc:62) >> ==26647== by 0x56D116: String::allocBuffer(unsigned long) (String.cc:56) >> ==26647== by 0x56D3A5: String::allocAndFill(char const*, int) (String.cc:131) >> ==26647== by 0x56D458: String::operator=(char const*) (String.cc:85) >> ==26647== by 0x530A4C: HttpHeaderEntry::HttpHeaderEntry(http_hdr_type, char const*, char const*) (HttpHeader.cc:1503) >> ==26647== by 0x53269C: HttpHeader::putStr(http_hdr_type, char const*) (HttpHeader.cc:1155) >> ==26647== by 0x536475: httpHeaderPutStrf (HttpHeaderTools.cc:126) >> ==26647== by 0x4D10B9: clientReplyContext::buildReplyHeader() (client_side_reply.cc:1426) >> ==26647== by 0x4D1DDA: clientReplyContext::sendMoreData(StoreIOBuffer) (client_side_reply.cc:2117) >> ==26647== by 0x57743C: store_client::callback(long, bool) (store_client.cc:165) >> ==26647== >> ==26647== Invalid read of size 8 >> ==26647== at 0x6CE3D2D: __strspn_sse42 (in /lib64/libc-2.12.so) >> ==26647== by 0x535F79: strListGetItem (HttpHeaderTools.cc:259) >> ==26647== by 0x51FA07: httpMakeVaryMark (http.cc:538) >> ==26647== by 0x4BB11A: varyEvaluateMatch (client_side.cc:3812) >> ==26647== by 0x4D44B2: clientReplyContext::cacheHit(StoreIOBuffer) (client_side_reply.cc:483) >> ==26647== by 0x57743C: store_client::callback(long, bool) (store_client.cc:165) >> ==26647== by 0x5778A4: store_client::scheduleMemRead() (store_client.cc:454) >> ==26647== by 0x577CEA: store_client::doCopy(StoreEntry*) (store_client.cc:383) >> ==26647== by 0x577FA5: storeClientCopy2(StoreEntry*, store_client*) (store_client.cc:337) >> ==26647== by 0x4D323B: clientReplyContext::doGetMoreData() (client_side_reply.cc:1751) >> ==26647== by 0x4D5614: ClientHttpRequest::httpStart() (client_side_request.cc:1358) >> ==26647== by 0x4D770A: ClientHttpRequest::doCallouts() (client_side_request.cc:1604) >> ==26647== Address 0x19f09418 is 0 bytes after a block of size 40 alloc'd >> ==26647== at 0x4C25A28: calloc (vg_replace_malloc.c:467) >> ==26647== by 0x65B441: xcalloc (xalloc.cc:75) >> ==26647== by 0x657B99: MemPoolMalloc::allocate() (MemPoolMalloc.cc:62) >> ==26647== by 0x56D116: String::allocBuffer(unsigned long) (String.cc:56) >> ==26647== by 0x56D3A5: String::allocAndFill(char const*, int) (String.cc:131) >> ==26647== by 0x56D458: String::operator=(char const*) (String.cc:85) >> ==26647== by 0x530A4C: HttpHeaderEntry::HttpHeaderEntry(http_hdr_type, char const*, char const*) (HttpHeader.cc:1503) >> ==26647== by 0x530BC9: HttpHeaderEntry::clone() const (HttpHeader.cc:1614) >> ==26647== by 0x532DDF: HttpHeader::append(HttpHeader const*) (HttpHeader.cc:484) >> ==26647== by 0x53A45F: HttpReply::clone() const (HttpReply.cc:615) >> ==26647== by 0x4D1A1B: clientReplyContext::cloneReply() (client_side_reply.cc:1515) >> ==26647== by 0x4D1DDA: clientReplyContext::sendMoreData(StoreIOBuffer) (client_side_reply.cc:2117) >> ==26647== >> ==26647== Invalid read of size 8 >> ==26647== at 0x6CE3D2D: __strspn_sse42 (in /lib64/libc-2.12.so) >> ==26647== by 0x535F79: strListGetItem (HttpHeaderTools.cc:259) >> ==26647== by 0x5705E4: StoreEntry::hasOneOfEtags(String const&, bool) const (store.cc:1981) >> ==26647== by 0x570741: StoreEntry::hasIfNoneMatchEtag(HttpRequest const&) const (store.cc:1966) >> ==26647== by 0x4D34EA: clientReplyContext::processConditional(StoreIOBuffer&) (client_side_reply.cc:711) >> ==26647== by 0x4D47B0: clientReplyContext::cacheHit(StoreIOBuffer) (client_side_reply.cc:571) >> ==26647== by 0x57743C: store_client::callback(long, bool) (store_client.cc:165) >> ==26647== by 0x5778A4: store_client::scheduleMemRead() (store_client.cc:454) >> ==26647== by 0x577CEA: store_client::doCopy(StoreEntry*) (store_client.cc:383) >> ==26647== by 0x577FA5: storeClientCopy2(StoreEntry*, store_client*) (store_client.cc:337) >> ==26647== by 0x4D323B: clientReplyContext::doGetMoreData() (client_side_reply.cc:1751) >> ==26647== by 0x4D45D5: clientReplyContext::cacheHit(StoreIOBuffer) (client_side_reply.cc:504) >> ==26647== Address 0x104c8748 is 0 bytes after a block of size 40 alloc'd >> ==26647== at 0x4C25A28: calloc (vg_replace_malloc.c:467) >> ==26647== by 0x65B441: xcalloc (xalloc.cc:75) >> ==26647== by 0x657B99: MemPoolMalloc::allocate() (MemPoolMalloc.cc:62) >> ==26647== by 0x56D116: String::allocBuffer(unsigned long) (String.cc:56) >> ==26647== by 0x56D3A5: String::allocAndFill(char const*, int) (String.cc:131) >> ==26647== by 0x56D458: String::operator=(char const*) (String.cc:85) >> ==26647== by 0x530A4C: HttpHeaderEntry::HttpHeaderEntry(http_hdr_type, char const*, char const*) (HttpHeader.cc:1503) >> ==26647== by 0x530BC9: HttpHeaderEntry::clone() const (HttpHeader.cc:1614) >> ==26647== by 0x51FE06: HttpStateData::httpBuildRequestHeader(HttpRequest*, StoreEntry*, RefCount<AccessLogEntry> const&, HttpHeader*, _http_state_flags) (http.cc:1954) >> ==26647== by 0x520D71: HttpStateData::buildRequestPrefix(MemBuf*) (http.cc:2007) >> ==26647== by 0x5259B9: HttpStateData::sendRequest() (http.cc:2112) >> ==26647== by 0x525E08: HttpStateData::start() (http.cc:2165) >> ==26647== >> >> >> >>
