Search squid archive

Re: problems with ssl_crtd

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 20/09/12 12:58, Ahmed Talha Khan wrote:
> Hey Guy, All
> 
> I have started facing a very similar issue now.I have been using
> squid-3.HEAD-20120421-r12120 for about 5 months without any issues.
> Suddenly from yesterday ive started getting crahses in ssl_crtd
> process.
> 
> 
> In my case i am the only user but i observe that the behaviour is
> random. Sometimes it crashes and sometimes it works. Different https
> pages give the crash. Even non https pages have caused the crash.
> 
>  These occur especially on google https pages like docs,mail,calender etc..
> 
> The signing cert is also ok and has NOT expired.
> 
> 
> My squid conf looks like this:
> *******************************************************
> sslproxy_cert_error allow all
> 
> sslcrtd_program /usr/local/squid-3.3/libexec/ssl_crtd -s
> /usr/local/squid-3.3/var/lib/ssl_db -M 4MB
> sslcrtd_children 5
> 
> http_port 192.168.8.134:3128 ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> cert=/home/asif/squid/www.sample.com.pem
> key=/home/asif/squid/www.sample.com.pem
> 
> http_port 192.168.8.134:8080
> 
> https_port 192.168.8.134:3129 ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> cert=/home/asif/squid/www.sample.com.pem
> key=/home/asif/squid/www.sample.com.pem
> *******************************************************
> 
> The ssl_db directory is initialized properly with correct permissions.
> 
> ***********************************************************
> [talha@localhost lib]$ pwd
> /usr/local/squid-3.3/var/lib
> 
> [talha@localhost lib]$ ls -al
> total 24
> drwxrwxrwx 3 root   root  4096 Sep 20 15:31 .
> drwxrwxrwx 6 root   root  4096 Sep 20 15:05 ..
> drwxrwxrwx 3 nobody talha 4096 Sep 20 15:31 ssl_db
> 
> The size file also has some values in it and cert generation also
> seems to work but suddenly it all crashes .
> **************************************************************
> 
> 
> 
> 2012/09/20 14:57:45| Starting Squid Cache version
> 3.HEAD-20120425-r12120 for x86_64-unknown-linux-gnu...
> 2012/09/20 14:57:45| Process ID 23826
> 2012/09/20 14:57:45| Process Roles: master worker
> 2012/09/20 14:57:45| With 1024 file descriptors available
> 2012/09/20 14:57:45| Initializing IP Cache...
> 2012/09/20 14:57:45| DNS Socket created at [::], FD 5
> 2012/09/20 14:57:45| DNS Socket created at 0.0.0.0, FD 6
> 2012/09/20 14:57:45| Adding nameserver 192.168.8.1 from /etc/resolv.conf
> 2012/09/20 14:57:45| Adding domain localdomain from /etc/resolv.conf
> 2012/09/20 14:57:45| helperOpenServers: Starting 5/5 'ssl_crtd' processes
> 2012/09/20 14:57:45| Logfile: opening log
> daemon:/usr/local/squid-3.3/var/logs/access.log
> 2012/09/20 14:57:45| Logfile Daemon: opening log
> /usr/local/squid-3.3/var/logs/access.log
> 2012/09/20 14:57:45| Logfile: opening log /usr/local/squid-3.3/var/logs/icap-log
> 2012/09/20 14:57:45| WARNING: log parameters now start with a module
> name. Use 'stdio:/usr/local/squid-3.3/var/logs/icap-log'
> 
> 
> 2012/09/20 14:57:45| Store logging disabled
> 2012/09/20 14:57:45| Swap maxSize 0 + 262144 KB, estimated 20164 objects
> 2012/09/20 14:57:45| Target number of buckets: 1008
> 2012/09/20 14:57:45| Using 8192 Store buckets
> 2012/09/20 14:57:45| Max Mem  size: 262144 KB
> 2012/09/20 14:57:45| Max Swap size: 0 KB
> 2012/09/20 14:57:45| Using Least Load store dir selection
> 2012/09/20 14:57:45| Set Current Directory to /usr/local/squid-3.3/var/cache
> 2012/09/20 14:57:45| Loaded Icons.
> 2012/09/20 14:57:45| HTCP Disabled.
> 2012/09/20 14:57:45| /usr/local/squid-3.3/var/run/squid.pid: (13)
> Permission denied
> 2012/09/20 14:57:45| WARNING: Could not write pid file
> 2012/09/20 14:57:45| Squid plugin modules loaded: 0
> 2012/09/20 14:57:45| Adaptation support is on
> 2012/09/20 14:57:45| Accepting SSL bumped HTTP Socket connections at
> local=192.168.8.134:3128 remote=[::] FD 20 flags=9
> 2012/09/20 14:57:45| Accepting HTTP Socket connections at
> local=192.168.8.134:8080 remote=[::] FD 21 flags=9
> 2012/09/20 14:57:45| Accepting SSL bumped HTTPS Socket connections at
> local=192.168.8.134:3129 remote=[::] FD 22 flags=9
> 2012/09/20 14:57:46| storeLateRelease: released 0 objects
> 
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/20 14:58:23| WARNING: ssl_crtd #2 exited
> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5)
> 
> 2012/09/20 14:58:23| Starting new helpers
> 2012/09/20 14:58:23| helperOpenServers: Starting 1/5 'ssl_crtd' processes
> 2012/09/20 14:58:23| client_side.cc(3478) sslCrtdHandleReply:
> "ssl_crtd" helper return <NULL> reply
> (ssl_crtd): Cannot create ssl certificate or private key.
> 
> 2012/09/20 14:58:23| WARNING: ssl_crtd #1 exited
> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5)
> 2012/09/20 14:58:23| storeDirWriteCleanLogs: Starting...
> 2012/09/20 14:58:23|   Finished.  Wrote 0 entries.
> 2012/09/20 14:58:23|   Took 0.00 seconds (  0.00 entries/sec).
> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
> 
> Squid Cache (Version 3.HEAD-20120425-r12120): Terminated abnormally.
> CPU Usage: 0.355 seconds = 0.289 user + 0.066 sys
> Maximum Resident Size: 71104 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
>     total space in arena:   11924 KB
>     Ordinary blocks:        11818 KB     49 blks
>     Small blocks:               0 KB      0 blks
>     Holding blocks:           664 KB      2 blks
>     Free Small blocks:          0 KB
>     Free Ordinary blocks:     105 KB
> 
> 
> 
> 
> 
> 
> On Thu, Sep 20, 2012 at 2:52 PM, Linos <info@xxxxxxxx> wrote:
>> On 19/09/12 16:46, Guy Helmer wrote:
>>>>
>>>> Thanks for reply.
>>>>
>>>> i checked the squid_ssl_db/size because i found the empty file problem searching
>>>> for my own problem in the mailing list, it's ok in my host, the file have the
>>>> content "139264" right now.
>>>>
>>>> I can't found the core file, do i need to do something for it to generate? maybe
>>>> a configure script option or squid.conf change to activate it?
>>>>
>>>> Regards,
>>>> Miguel Angel.
>>>
>>> I have
>>>
>>> coredump_dir /var/log/squid
>>>
>>> to get coredumps in my /var/log/squid directory. Now that I think about it, I don't remember if this works for ssl_crtd though -- seems like I have had to start "gdb ssl_crtd" and then attach to one of the ssl_crtd processes, then generate HTTPS traffic to trigger the request to ssl_crtd and get a backtrace when ssl_crtd gets the segfault signal…
>>>
>>> Guy
>>>
>>
>> Hi,
>>    i have been trying to debug with gdb attaching existing process, the strange
>> it's that ssl_ctrd seems to exit normally in this test, here you have it (sorry
>> for the spanish locale, i will use english next time, the only file with symbols
>> it's ssl_crtd itself):
>>
>> --------------------------------------------------------------------------------
>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Para las instrucciones de informe de errores, vea:
>> <http://bugs.launchpad.net/gdb-linaro/>.
>> (gdb) attach 10495
>> Adjuntando a process 10495
>> Leyendo símbolos desde /usr/lib/squid3/ssl_crtd...Leyendo símbolos desde
>> /usr/lib/debug/usr/lib/squid3/ssl_crtd...hecho.
>> hecho.
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libc.so.6
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libz.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libm.so.6
>> Leyendo símbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron símbolos
>> de depuración)hecho.
>> Símbolos cargados para /lib64/ld-linux-x86-64.so.2
>> 0x00007f3ef414f0a0 in read () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) continue
>> Continuando.
>> [Inferior 1 (process 10495) exited normally]
>> (gdb) bt
>> No stack.
>>
>> --------------------------------------------------------------------------------
>>
>> I have tried attaching to squid3 process itself and i have received a signal here:
>>
>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
>> Copyright (C) 2012 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Para las instrucciones de informe de errores, vea:
>> <http://bugs.launchpad.net/gdb-linaro/>.
>> (gdb) attach 10732
>> Adjuntando a process 10732
>> Leyendo símbolos desde /usr/sbin/squid3...coLeyendo símbolos desde
>> /usr/lib/debug/usr/sbin/squid3...ntinue
>> hecho.
>> hecho.
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libpthread.so.0...(no se
>> encontraron símbolos de depuración)hecho.
>> [Depuración de hilo usando libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Símbolos cargados para /lib/x86_64-linux-gnu/libpthread.so.0
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libxml2.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libexpat.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libexpat.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libssl.so.0.9.8...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libssl.so.0.9.8
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5.so.3
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcom_err.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libcom_err.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcap.so.2...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libcap.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/librt.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/librt.so.1
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libltdl.so.7...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libltdl.so.7
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libm.so.6
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libc.so.6
>> Leyendo símbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron símbolos
>> de depuración)hecho.
>> Símbolos cargados para /lib64/ld-linux-x86-64.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libz.so.1
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libkeyutils.so.1...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libkeyutils.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libresolv.so.2...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libresolv.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_files.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_files.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_compat.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_compat.so.2
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnsl.so.1...(no se encontraron
>> símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libnsl.so.1
>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_nis.so.2...(no se
>> encontraron símbolos de depuración)hecho.
>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_nis.so.2
>> 0x00007f7d6243dac8 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) continue
>> Continuando.
>>
>> Program received signal SIGPIPE, Broken pipe.
>> 0x00007f7d647becb0 in __write_nocancel () from /lib/x86_64-linux-gnu/libpthread.so.0
>> (gdb) bt
>> #0  0x00007f7d647becb0 in __write_nocancel () from
>> /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00007f7d63d075c5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>> #2  0x00007f7d63d05247 in BIO_write () from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8
>> #3  0x00007f7d63ffafc4 in ssl3_write_pending () from
>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>> #4  0x00007f7d63ffc853 in ssl3_dispatch_alert () from
>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>> #5  0x00007f7d63ff9442 in ssl3_shutdown () from
>> /lib/x86_64-linux-gnu/libssl.so.0.9.8
>> #6  0x00007f7d64e8f0f4 in AsyncCall::make (this=0x7f7d687eb390) at AsyncCall.cc:36
>> #7  0x00007f7d64e92117 in AsyncCallQueue::fireNext (this=<optimized out>) at
>> AsyncCallQueue.cc:54
>> #8  0x00007f7d64e92270 in AsyncCallQueue::fire (this=0x7f7d66f5f2c0) at
>> AsyncCallQueue.cc:40
>> #9  0x00007f7d64d7c494 in EventLoop::runOnce (this=0x7fff630b3e60) at
>> EventLoop.cc:131
>> #10 0x00007f7d64d7c568 in EventLoop::run (this=0x7fff630b3e60) at EventLoop.cc:95
>> #11 0x00007f7d64ddc039 in SquidMain (argc=<optimized out>, argv=<optimized out>)
>> at main.cc:1500
>> #12 0x00007f7d64d10b76 in SquidMainSafe (argv=<optimized out>, argc=<optimized
>> out>) at main.cc:1215
>> #13 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1207
>>
>> Any ideas what's going on with this information? Thansk!
>>
>> Regards,
>> Miguel Angel.
> 
> 
> 

Hi Ahmed,
	did you found a way to fix or workaround this? i can't get a backtrace with the
information needed and no matter what i try it keeps failing, i had to disable
ssl and my users are a bit upset.

Regards,
Miguel Angel.


[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux