Linos, I have not debugged the issue yet. Will post results when do it. Can anyone provide with the FATAL patch kindly? -talha On Mon, Sep 24, 2012 at 9:47 PM, Linos <info@xxxxxxxx> wrote: > On 24/09/12 12:52, Amos Jeffries wrote: >> On 24/09/2012 8:44 p.m., Linos wrote: >>> On 20/09/12 12:58, Ahmed Talha Khan wrote: >>>> Hey Guy, All >>>> >>>> I have started facing a very similar issue now.I have been using >>>> squid-3.HEAD-20120421-r12120 for about 5 months without any issues. >>>> Suddenly from yesterday ive started getting crahses in ssl_crtd >>>> process. >>>> >>>> >>>> In my case i am the only user but i observe that the behaviour is >>>> random. Sometimes it crashes and sometimes it works. Different https >>>> pages give the crash. Even non https pages have caused the crash. >>>> >>>> These occur especially on google https pages like docs,mail,calender etc.. >>>> >>>> The signing cert is also ok and has NOT expired. >>>> >>>> >>>> My squid conf looks like this: >>>> ******************************************************* >>>> sslproxy_cert_error allow all >>>> >>>> sslcrtd_program /usr/local/squid-3.3/libexec/ssl_crtd -s >>>> /usr/local/squid-3.3/var/lib/ssl_db -M 4MB >>>> sslcrtd_children 5 >>>> >>>> http_port 192.168.8.134:3128 ssl-bump generate-host-certificates=on >>>> dynamic_cert_mem_cache_size=4MB >>>> cert=/home/asif/squid/www.sample.com.pem >>>> key=/home/asif/squid/www.sample.com.pem >>>> >>>> http_port 192.168.8.134:8080 >>>> >>>> https_port 192.168.8.134:3129 ssl-bump generate-host-certificates=on >>>> dynamic_cert_mem_cache_size=4MB >>>> cert=/home/asif/squid/www.sample.com.pem >>>> key=/home/asif/squid/www.sample.com.pem >>>> ******************************************************* >>>> >>>> The ssl_db directory is initialized properly with correct permissions. >>>> >>>> *********************************************************** >>>> [talha@localhost lib]$ pwd >>>> /usr/local/squid-3.3/var/lib >>>> >>>> [talha@localhost lib]$ ls -al >>>> total 24 >>>> drwxrwxrwx 3 root root 4096 Sep 20 15:31 . >>>> drwxrwxrwx 6 root root 4096 Sep 20 15:05 .. >>>> drwxrwxrwx 3 nobody talha 4096 Sep 20 15:31 ssl_db >>>> >>>> The size file also has some values in it and cert generation also >>>> seems to work but suddenly it all crashes . >>>> ************************************************************** >>>> >>>> >>>> >>>> 2012/09/20 14:57:45| Starting Squid Cache version >>>> 3.HEAD-20120425-r12120 for x86_64-unknown-linux-gnu... >>>> 2012/09/20 14:57:45| Process ID 23826 >>>> 2012/09/20 14:57:45| Process Roles: master worker >>>> 2012/09/20 14:57:45| With 1024 file descriptors available >>>> 2012/09/20 14:57:45| Initializing IP Cache... >>>> 2012/09/20 14:57:45| DNS Socket created at [::], FD 5 >>>> 2012/09/20 14:57:45| DNS Socket created at 0.0.0.0, FD 6 >>>> 2012/09/20 14:57:45| Adding nameserver 192.168.8.1 from /etc/resolv.conf >>>> 2012/09/20 14:57:45| Adding domain localdomain from /etc/resolv.conf >>>> 2012/09/20 14:57:45| helperOpenServers: Starting 5/5 'ssl_crtd' processes >>>> 2012/09/20 14:57:45| Logfile: opening log >>>> daemon:/usr/local/squid-3.3/var/logs/access.log >>>> 2012/09/20 14:57:45| Logfile Daemon: opening log >>>> /usr/local/squid-3.3/var/logs/access.log >>>> 2012/09/20 14:57:45| Logfile: opening log /usr/local/squid-3.3/var/logs/icap-log >>>> 2012/09/20 14:57:45| WARNING: log parameters now start with a module >>>> name. Use 'stdio:/usr/local/squid-3.3/var/logs/icap-log' >>>> >>>> >>>> 2012/09/20 14:57:45| Store logging disabled >>>> 2012/09/20 14:57:45| Swap maxSize 0 + 262144 KB, estimated 20164 objects >>>> 2012/09/20 14:57:45| Target number of buckets: 1008 >>>> 2012/09/20 14:57:45| Using 8192 Store buckets >>>> 2012/09/20 14:57:45| Max Mem size: 262144 KB >>>> 2012/09/20 14:57:45| Max Swap size: 0 KB >>>> 2012/09/20 14:57:45| Using Least Load store dir selection >>>> 2012/09/20 14:57:45| Set Current Directory to /usr/local/squid-3.3/var/cache >>>> 2012/09/20 14:57:45| Loaded Icons. >>>> 2012/09/20 14:57:45| HTCP Disabled. >>>> 2012/09/20 14:57:45| /usr/local/squid-3.3/var/run/squid.pid: (13) >>>> Permission denied >>>> 2012/09/20 14:57:45| WARNING: Could not write pid file >>>> 2012/09/20 14:57:45| Squid plugin modules loaded: 0 >>>> 2012/09/20 14:57:45| Adaptation support is on >>>> 2012/09/20 14:57:45| Accepting SSL bumped HTTP Socket connections at >>>> local=192.168.8.134:3128 remote=[::] FD 20 flags=9 >>>> 2012/09/20 14:57:45| Accepting HTTP Socket connections at >>>> local=192.168.8.134:8080 remote=[::] FD 21 flags=9 >>>> 2012/09/20 14:57:45| Accepting SSL bumped HTTPS Socket connections at >>>> local=192.168.8.134:3129 remote=[::] FD 22 flags=9 >>>> 2012/09/20 14:57:46| storeLateRelease: released 0 objects >>>> >>>> (ssl_crtd): Cannot create ssl certificate or private key. >>>> 2012/09/20 14:58:23| WARNING: ssl_crtd #2 exited >>>> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5) >>>> >>>> 2012/09/20 14:58:23| Starting new helpers >>>> 2012/09/20 14:58:23| helperOpenServers: Starting 1/5 'ssl_crtd' processes >>>> 2012/09/20 14:58:23| client_side.cc(3478) sslCrtdHandleReply: >>>> "ssl_crtd" helper return <NULL> reply >>>> (ssl_crtd): Cannot create ssl certificate or private key. >>>> >>>> 2012/09/20 14:58:23| WARNING: ssl_crtd #1 exited >>>> 2012/09/20 14:58:23| Too few ssl_crtd processes are running (need 1/5) >>>> 2012/09/20 14:58:23| storeDirWriteCleanLogs: Starting... >>>> 2012/09/20 14:58:23| Finished. Wrote 0 entries. >>>> 2012/09/20 14:58:23| Took 0.00 seconds ( 0.00 entries/sec). >>>> FATAL: The ssl_crtd helpers are crashing too rapidly, need help! >>>> >>>> Squid Cache (Version 3.HEAD-20120425-r12120): Terminated abnormally. >>>> CPU Usage: 0.355 seconds = 0.289 user + 0.066 sys >>>> Maximum Resident Size: 71104 KB >>>> Page faults with physical i/o: 0 >>>> Memory usage for squid via mallinfo(): >>>> total space in arena: 11924 KB >>>> Ordinary blocks: 11818 KB 49 blks >>>> Small blocks: 0 KB 0 blks >>>> Holding blocks: 664 KB 2 blks >>>> Free Small blocks: 0 KB >>>> Free Ordinary blocks: 105 KB >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Sep 20, 2012 at 2:52 PM, Linos wrote: >>>>> On 19/09/12 16:46, Guy Helmer wrote: >>>>>>> Thanks for reply. >>>>>>> >>>>>>> i checked the squid_ssl_db/size because i found the empty file problem >>>>>>> searching >>>>>>> for my own problem in the mailing list, it's ok in my host, the file have the >>>>>>> content "139264" right now. >>>>>>> >>>>>>> I can't found the core file, do i need to do something for it to generate? >>>>>>> maybe >>>>>>> a configure script option or squid.conf change to activate it? >>>>>>> >>>>>>> Regards, >>>>>>> Miguel Angel. >>>>>> I have >>>>>> >>>>>> coredump_dir /var/log/squid >>>>>> >>>>>> to get coredumps in my /var/log/squid directory. Now that I think about it, >>>>>> I don't remember if this works for ssl_crtd though -- seems like I have had >>>>>> to start "gdb ssl_crtd" and then attach to one of the ssl_crtd processes, >>>>>> then generate HTTPS traffic to trigger the request to ssl_crtd and get a >>>>>> backtrace when ssl_crtd gets the segfault signal… >>>>>> >>>>>> Guy >>>>>> >>>>> Hi, >>>>> i have been trying to debug with gdb attaching existing process, the >>>>> strange >>>>> it's that ssl_ctrd seems to exit normally in this test, here you have it (sorry >>>>> for the spanish locale, i will use english next time, the only file with >>>>> symbols >>>>> it's ssl_crtd itself): >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> >>>>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04 >>>>> Copyright (C) 2012 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >>>>> and "show warranty" for details. >>>>> This GDB was configured as "x86_64-linux-gnu". >>>>> Para las instrucciones de informe de errores, vea: >>>>> <http://bugs.launchpad.net/gdb-linaro/>. >>>>> (gdb) attach 10495 >>>>> Adjuntando a process 10495 >>>>> Leyendo símbolos desde /usr/lib/squid3/ssl_crtd...Leyendo símbolos desde >>>>> /usr/lib/debug/usr/lib/squid3/ssl_crtd...hecho. >>>>> hecho. >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libc.so.6 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libz.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libm.so.6 >>>>> Leyendo símbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron >>>>> símbolos >>>>> de depuración)hecho. >>>>> Símbolos cargados para /lib64/ld-linux-x86-64.so.2 >>>>> 0x00007f3ef414f0a0 in read () from /lib/x86_64-linux-gnu/libc.so.6 >>>>> (gdb) continue >>>>> Continuando. >>>>> [Inferior 1 (process 10495) exited normally] >>>>> (gdb) bt >>>>> No stack. >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> >>>>> >>>>> I have tried attaching to squid3 process itself and i have received a signal >>>>> here: >>>>> >>>>> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04 >>>>> Copyright (C) 2012 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >>>>> and "show warranty" for details. >>>>> This GDB was configured as "x86_64-linux-gnu". >>>>> Para las instrucciones de informe de errores, vea: >>>>> <http://bugs.launchpad.net/gdb-linaro/>. >>>>> (gdb) attach 10732 >>>>> Adjuntando a process 10732 >>>>> Leyendo símbolos desde /usr/sbin/squid3...coLeyendo símbolos desde >>>>> /usr/lib/debug/usr/sbin/squid3...ntinue >>>>> hecho. >>>>> hecho. >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libpthread.so.0...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> [Depuración de hilo usando libthread_db enabled] >>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libxml2.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libexpat.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libexpat.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libssl.so.0.9.8...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libssl.so.0.9.8 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5.so.3 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcom_err.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libcom_err.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libcap.so.2...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libcap.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/librt.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/librt.so.1 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libltdl.so.7...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libltdl.so.7 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libm.so.6...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libm.so.6 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libgcc_s.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libgcc_s.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libc.so.6...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libc.so.6 >>>>> Leyendo símbolos desde /lib64/ld-linux-x86-64.so.2...(no se encontraron >>>>> símbolos >>>>> de depuración)hecho. >>>>> Símbolos cargados para /lib64/ld-linux-x86-64.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libdl.so.2...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libdl.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libz.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libz.so.1 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 >>>>> Leyendo símbolos desde /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libkeyutils.so.1...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libkeyutils.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libresolv.so.2...(no se >>>>> encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libresolv.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_files.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_files.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_compat.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_compat.so.2 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnsl.so.1...(no se encontraron >>>>> símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libnsl.so.1 >>>>> Leyendo símbolos desde /lib/x86_64-linux-gnu/libnss_nis.so.2...(no se >>>>> encontraron símbolos de depuración)hecho. >>>>> Símbolos cargados para /lib/x86_64-linux-gnu/libnss_nis.so.2 >>>>> 0x00007f7d6243dac8 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >>>>> (gdb) continue >>>>> Continuando. >>>>> >>>>> Program received signal SIGPIPE, Broken pipe. >>>>> 0x00007f7d647becb0 in __write_nocancel () from >>>>> /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> (gdb) bt >>>>> #0 0x00007f7d647becb0 in __write_nocancel () from >>>>> /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> #1 0x00007f7d63d075c5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 >>>>> #2 0x00007f7d63d05247 in BIO_write () from >>>>> /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 >>>>> #3 0x00007f7d63ffafc4 in ssl3_write_pending () from >>>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8 >>>>> #4 0x00007f7d63ffc853 in ssl3_dispatch_alert () from >>>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8 >>>>> #5 0x00007f7d63ff9442 in ssl3_shutdown () from >>>>> /lib/x86_64-linux-gnu/libssl.so.0.9.8 >>>>> #6 0x00007f7d64e8f0f4 in AsyncCall::make (this=0x7f7d687eb390) at >>>>> AsyncCall.cc:36 >>>>> #7 0x00007f7d64e92117 in AsyncCallQueue::fireNext (this=<optimized out>) at >>>>> AsyncCallQueue.cc:54 >>>>> #8 0x00007f7d64e92270 in AsyncCallQueue::fire (this=0x7f7d66f5f2c0) at >>>>> AsyncCallQueue.cc:40 >>>>> #9 0x00007f7d64d7c494 in EventLoop::runOnce (this=0x7fff630b3e60) at >>>>> EventLoop.cc:131 >>>>> #10 0x00007f7d64d7c568 in EventLoop::run (this=0x7fff630b3e60) at >>>>> EventLoop.cc:95 >>>>> #11 0x00007f7d64ddc039 in SquidMain (argc=<optimized out>, argv=<optimized >>>>> out>) >>>>> at main.cc:1500 >>>>> #12 0x00007f7d64d10b76 in SquidMainSafe (argv=<optimized out>, argc=<optimized >>>>> out>) at main.cc:1215 >>>>> #13 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1207 >>>>> >>>>> Any ideas what's going on with this information? Thansk! >>>>> >>>>> Regards, >>>>> Miguel Angel. >>>> >>>> >>> Hi Ahmed, >>> did you found a way to fix or workaround this? i can't get a backtrace >>> with the >>> information needed and no matter what i try it keeps failing, i had to disable >>> ssl and my users are a bit upset. >>> >>> Regards, >>> Miguel Angel. >> >> I've taken a quick look at these reports and tried to figure out if anything >> obvious is going on. >> >> From what I can see, there is really no reason why the helper should be exiting >> like this. Something it is doing produces an exception, fine, but the method of >> handling that exception is to exit instead of reporting the problem to Squid and >> continuing with further requests. >> >> As a workaround you can probably patch out the FATAL and let Squid continuously >> restart the helpers unless that spams the logs too much for your liking. >> >> Amos >> > > Thanks Amos, i commented out this section like you advised me and now i am > working normally, it's not a fix but it permits me keep using squid normally. > > Regards, > Miguel Angel. -- Regards, -Ahmed Talha Khan
