Hi Amos,
New backtrace - I hope this helps!
Core was generated by `(squid-1) -YC -f /etc/squid3/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00007f2f758a81b5 in raise () from /lib/libc.so.6
(gdb) bt full
#0 0x00007f2f758a81b5 in raise () from /lib/libc.so.6
No symbol table info available.
#1 0x00007f2f758aafc0 in abort () from /lib/libc.so.6
No symbol table info available.
#2 0x000000000054670f in xassert (msg=0x7bb62c "c", file=0x7ea5f8
"base/CbcPointer.h", line=147) at debug.cc:565
__FUNCTION__ = "xassert"
#3 0x00000000005279d1 in CbcPointer<ConnStateData>::operator->
(this=<value optimized out>) at base/CbcPointer.h:147
c = <value optimized out>
#4 0x000000000057238e in FwdState::initiateSSL (this=0x80f14ba8) at
forward.cc:827
hostname = 0x80e6d7e8 "secure.flashtalking.com"
isConnectRequest = <value optimized out>
peer = <value optimized out>
fd = 812
__FUNCTION__ = "initiateSSL"
peeked_cert = <value optimized out>
ssl = 0x940e87e0
sslContext = <value optimized out>
#5 0x00000000005725e3 in FwdState::connectDone (this=0x80f14ba8,
conn=..., status=<value optimized out>, xerrno=0) at forward.cc:895
__FUNCTION__ = "connectDone"
#6 0x00000000006a6f69 in AsyncCall::make (this=0x950cf990) at
AsyncCall.cc:32
__FUNCTION__ = "make"
#7 0x00000000006aa215 in AsyncCallQueue::fireNext (this=<value
optimized out>) at AsyncCallQueue.cc:52
call = {p_ = 0x950cf990}
__FUNCTION__ = "fireNext"
#8 0x00000000006aa3c0 in AsyncCallQueue::fire (this=0xfb53f0) at
AsyncCallQueue.cc:38
made = true
#9 0x00000000005633dc in EventLoop::runOnce (this=0x7fffd3a62b20) at
EventLoop.cc:132
sawActivity = false
waitingEngine = 0x7fffd3a62ba0
__FUNCTION__ = "runOnce"
#10 0x0000000000563518 in EventLoop::run (this=0x7fffd3a62b20) at
EventLoop.cc:96
No locals.
#11 0x00000000005d3a25 in SquidMain (argc=<value optimized out>,
argv=<value optimized out>) at main.cc:1520
WIN32_init_err = <value optimized out>
__FUNCTION__ = "SquidMain"
signalEngine = {<AsyncEngine> = {_vptr.AsyncEngine = 0x7cc770},
loop = @0x7fffd3a62b20}
store_engine = {<AsyncEngine> = {_vptr.AsyncEngine = 0x7cc7d0},
<No data fields>}
comm_engine = {<AsyncEngine> = {_vptr.AsyncEngine = 0xa78f30},
<No data fields>}
mainLoop = {errcount = 0, last_loop = false, engines =
{capacity = 16, count = 4, items = 0x1426140}, timeService =
0x7fffd3a62b90, primaryEngine = 0x7fffd3a62ba0, loop_delay = 0, error =
false, runOnceResult = false}
time_engine = {_vptr.TimeEngine = 0x7dbe90}
#12 0x00000000005d4213 in SquidMainSafe (argc=3051, argv=0xbeb) at
main.cc:1242
No locals.
#13 main (argc=3051, argv=0xbeb) at main.cc:1234
No locals.
We are also getting a lot of this sort of thing in the logs since I've
patched that Assert. Not sure If it's related.
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 13:22:57 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 13:22:57 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 13:22:57 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 13:22:57 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 13:22:57 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 13:22:57 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 13:22:57 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 13:22:57 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 14:21:49 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 14:21:49 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 14:21:49 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:49 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 14:21:50 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 14:21:50 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 14:21:50 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:50 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 14:21:50 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 14:21:50 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 14:21:50 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:50 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 14:21:50 kid1| Starting new ntlmauthenticator helpers...
2014/05/09 14:21:50 kid1| helperOpenServers: Starting 1/75 'ntlm_auth'
processes
2014/05/09 14:21:50 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:50 kid1| WARNING: Cannot run '/usr/bin/ntlm_auth' process.
2014/05/09 14:21:51 kid1| WARNING: Error Pages Missing Language: en-gb
2014/05/09 14:21:51 kid1| WARNING: Error Pages Missing Language: en
2014/05/09 14:21:51 kid1| Starting new ssl_crtd helpers...
2014/05/09 14:21:51 kid1| helperOpenServers: Starting 1/32 'ssl_crtd'
processes
2014/05/09 14:21:51 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:51 kid1| WARNING: Cannot run '/usr/lib/squid3/ssl_crtd'
process.
2014/05/09 14:21:51 kid1| Starting new ssl_crtd helpers...
2014/05/09 14:21:51 kid1| helperOpenServers: Starting 1/32 'ssl_crtd'
processes
2014/05/09 14:21:51 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:51 kid1| WARNING: Cannot run '/usr/lib/squid3/ssl_crtd'
process.
2014/05/09 14:21:51 kid1| Starting new ssl_crtd helpers...
2014/05/09 14:21:51 kid1| helperOpenServers: Starting 1/32 'ssl_crtd'
processes
2014/05/09 14:21:51 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:21:51 kid1| WARNING: Cannot run '/usr/lib/squid3/ssl_crtd'
process.
2014/05/09 14:39:16 kid1| helperOpenServers: Starting 10/70
'ext_wbinfo_group_acl' processes
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
2014/05/09 14:39:16 kid1| ipcCreate: fork: (12) Cannot allocate memory
2014/05/09 14:39:16 kid1| WARNING: Cannot run
'/usr/lib/squid3/ext_wbinfo_group_acl' process.
Thanks very much for looking at this!
Cheers
Alex
On 30/04/14 20:15, Amos Jeffries wrote:
On 1/05/2014 6:19 a.m., Alex Crow wrote:
Brilliant! I will try to apply this and see if we get more detail. Will
it apply to 3.2.x? I can't run 3.4.x in prod due to the CPU load issue -
and I only see the crash in prod, never managed to trigger it in a test.
Yes it should apply on 3.2, though I have not tested that.
Amos
Cheers
Alex
On 29/04/14 20:45, Amos Jeffries wrote:
On 30/04/2014 7:30 a.m., Alex Crow wrote:
dying from an unhandled exception: c
I just realised what is generating this is a "Must(c)". There are only
two of them Squid, but unfortunately in the generic and widely used
CbcPointer template.
Can you apply this patch please and see if we get a useful backtrace
next time:
http://www.squid-cache.org/Versions/v3/3.HEAD/changesets/squid-3-13386.patch
(this will also be in 3.4.5 to help with future issues hidden by the
same message).
Amos
