On Sep 19, 2012, at 5:44 AM, Linos <info@xxxxxxxx> wrote:
> Hi,
> i have been using Squid squid-3.2.0.17-20120527-r11561 in an Ubuntu Server
> 12.04 some time with ssl-bump without problems for a year, the ca cert expired
> some days ago and with the new ca cert i installed squid 3.2.1 stable.
>
> Now the proxy exists every time 10 or more users use https at the same time,
> it's pretty strange, i have tried to downgrade to the old squid version but i
> can't get the proxy to be stable no matter if using new or old version, i have
> tried to recreate other cert just in case, same problem, i recreated too
> squid_ssl_db and cache_dir, no matter what i do it keeps crashing, the cache log
> read as this:
>
>
> ------------------------------------------------------------------------------
> 2012/09/19 11:58:00| Starting Squid Cache version 3.2.1 for x86_64-pc-linux-gnu...
> 2012/09/19 11:58:00| Process ID 30077
> 2012/09/19 11:58:00| Process Roles: master worker
> 2012/09/19 11:58:00| With 65535 file descriptors available
> 2012/09/19 11:58:00| Initializing IP Cache...
> 2012/09/19 11:58:00| DNS Socket created at [::], FD 4
> 2012/09/19 11:58:00| DNS Socket created at 0.0.0.0, FD 5
> 2012/09/19 11:58:00| Adding nameserver 80.58.61.250 from squid.conf
> 2012/09/19 11:58:00| Adding nameserver 8.8.8.8 from squid.conf
> 2012/09/19 11:58:00| helperOpenServers: Starting 5/10 'ssl_crtd' processes
> 2012/09/19 11:58:00| helperOpenServers: Starting 5/20 'request_body_max_size.sh'
> processes
> 2012/09/19 11:58:00| Logfile: opening log daemon:/var/log/squid3/access.log
> 2012/09/19 11:58:00| Logfile Daemon: opening log /var/log/squid3/access.log
> 2012/09/19 11:58:00| Unlinkd pipe opened on FD 31
> 2012/09/19 11:58:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
> 2012/09/19 11:58:00| Store logging disabled
> 2012/09/19 11:58:00| Swap maxSize 15360000 + 262144 KB, estimated 312442 objects
> 2012/09/19 11:58:00| Target number of buckets: 15622
> 2012/09/19 11:58:00| Using 16384 Store buckets
> 2012/09/19 11:58:00| Max Mem size: 262144 KB
> 2012/09/19 11:58:00| Max Swap size: 15360000 KB
> 2012/09/19 11:58:00| Rebuilding storage in /mnt/squid/squid3 (clean log)
> 2012/09/19 11:58:00| Using Least Load store dir selection
> 2012/09/19 11:58:00| Set Current Directory to /mnt/squid/squid3
> 2012/09/19 11:58:00| Loaded Icons.
> 2012/09/19 11:58:00| HTCP Disabled.
> 2012/09/19 11:58:00| Squid plugin modules loaded: 0
> 2012/09/19 11:58:00| Adaptation support is off.
> 2012/09/19 11:58:00| Accepting NAT intercepted HTTP Socket connections at
> local=0.0.0.0:3128 remote=[::] FD 36 flags=41
> 2012/09/19 11:58:00| Accepting SSL bumped HTTP Socket connections at
> local=[::]:3150 remote=[::] FD 37 flags=9
> 2012/09/19 11:58:00| Store rebuilding is 16.55% complete
> 2012/09/19 11:58:00| Done reading /mnt/squid/squid3 swaplog (24167 entries)
> 2012/09/19 11:58:00| Finished rebuilding storage from disk.
> 2012/09/19 11:58:00| 24167 Entries scanned
> 2012/09/19 11:58:00| 0 Invalid entries.
> 2012/09/19 11:58:00| 0 With invalid flags.
> 2012/09/19 11:58:00| 24167 Objects loaded.
> 2012/09/19 11:58:00| 0 Objects expired.
> 2012/09/19 11:58:00| 0 Objects cancelled.
> 2012/09/19 11:58:00| 0 Duplicate URLs purged.
> 2012/09/19 11:58:00| 0 Swapfile clashes avoided.
> 2012/09/19 11:58:00| Took 0.12 seconds (204025.29 objects/sec).
> 2012/09/19 11:58:00| Beginning Validation Procedure
> 2012/09/19 11:58:00| Completed Validation Procedure
> 2012/09/19 11:58:00| Validated 24167 Entries
> 2012/09/19 11:58:00| store_swap_size = 732468.00 KB
> 2012/09/19 11:58:01| storeLateRelease: released 0 objects
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/19 12:03:20| WARNING: ssl_crtd #1 exited
> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
> 2012/09/19 12:03:20| Starting new helpers
> 2012/09/19 12:03:20| helperOpenServers: Starting 1/10 'ssl_crtd' processes
> 2012/09/19 12:03:20| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper
> return <NULL> reply
> (ssl_crtd): Cannot create ssl certificate or private key.
> 2012/09/19 12:03:20| WARNING: ssl_crtd #2 exited
> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
> 2012/09/19 12:03:20| Closing HTTP port 0.0.0.0:3128
> 2012/09/19 12:03:20| Closing HTTP port [::]:3150
> 2012/09/19 12:03:20| storeDirWriteCleanLogs: Starting...
> 2012/09/19 12:03:20| Finished. Wrote 24195 entries.
> 2012/09/19 12:03:20| Took 0.02 seconds (1321120.45 entries/sec).
> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
>
> Squid Cache (Version 3.2.1): Terminated abnormally.
> CPU Usage: 1.896 seconds = 0.740 user + 1.156 sys
> Maximum Resident Size: 144640 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> total space in arena: 18900 KB
> Ordinary blocks: 18674 KB 54 blks
> Small blocks: 0 KB 1 blks
> Holding blocks: 37552 KB 9 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 225 KB
> Total in use: 56226 KB 297%
> Total free: 225 KB 1%
>
>
> ------------------------------------------------------------------------------
> configure and kernel versions:
>
> kernel: 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64
> x86_64 x86_64 GNU/Linux
>
> Squid Cache: Version 3.2.1
> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
> '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode'
> '--disable-dependency-tracking' '--disable-silent-rules'
> '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
> '--mandir=/usr/share/man' '--with-cppunit-basedir=/usr' '--enable-inline'
> '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd'
> '--enable-removal-policies=lru,heap' '--enable-delay-pools'
> '--enable-cache-digests' '--enable-underscores' '--enable-icap-client'
> '--enable-follow-x-forwarded-for' '--enable-auth-basic' '--enable-auth-digest'
> '--enable-auth-ntlm' '--enable-auth-negotiate'
> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
> '--enable-ntlm-auth-helpers=smb_lm,'
> '--enable-digest-auth-helpers=ldap,password'
> '--enable-negotiate-auth-helpers=squid_kerb_auth'
> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
> '--enable-arp-acl' '--enable-esi' '--enable-zph-qos' '--enable-wccpv2'
> '--disable-translation' '--with-logdir=/var/log/squid3'
> '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536'
> '--with-large-files' '--with-default-user=proxy' '--enable-ssl'
> '--enable-ssl-crtd' '--disable-epoll' '--enable-linux-netfilter'
> 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector
> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
> 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now'
> 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector
> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>
>
> ------------------------------------------------------------------------------
> i get this in dmesg:
>
> [3433312.743391] init: squid3 main process (29801) terminated with status 1
> [3433417.181960] ssl_crtd[29848]: segfault at 0 ip 00007f2ee2494ac5 sp
> 00007fff51dbd260 error 4 in ssl_crtd[7f2ee248b000+10000]
> [3433417.328898] init: squid3 main process (29847) terminated with status 1
> [3433557.765878] init: squid3 main process (29877) killed by KILL signal
> [3433944.030777] init: squid3 main process (30077) terminated with status 1
>
>
> ------------------------------------------------------------------------------
>
OK, ssl_crtd is dying with a segfault when it is handling a request.
What is in the squid_ssl_db/size file? If it is empty (0 bytes), that would cause a crash in older (pre-3.2.1) squid.
Otherwise, is there an ssl_crtd core file in the squid log directory? A stack trace from it might help diagnose the problem.
Guy
