On 19/09/12 15:30, Guy Helmer wrote:
> On Sep 19, 2012, at 5:44 AM, Linos <info@xxxxxxxx> wrote:
>
>> Hi,
>> i have been using Squid squid-3.2.0.17-20120527-r11561 in an Ubuntu Server
>> 12.04 some time with ssl-bump without problems for a year, the ca cert expired
>> some days ago and with the new ca cert i installed squid 3.2.1 stable.
>>
>> Now the proxy exists every time 10 or more users use https at the same time,
>> it's pretty strange, i have tried to downgrade to the old squid version but i
>> can't get the proxy to be stable no matter if using new or old version, i have
>> tried to recreate other cert just in case, same problem, i recreated too
>> squid_ssl_db and cache_dir, no matter what i do it keeps crashing, the cache log
>> read as this:
>>
>>
>> ------------------------------------------------------------------------------
>> 2012/09/19 11:58:00| Starting Squid Cache version 3.2.1 for x86_64-pc-linux-gnu...
>> 2012/09/19 11:58:00| Process ID 30077
>> 2012/09/19 11:58:00| Process Roles: master worker
>> 2012/09/19 11:58:00| With 65535 file descriptors available
>> 2012/09/19 11:58:00| Initializing IP Cache...
>> 2012/09/19 11:58:00| DNS Socket created at [::], FD 4
>> 2012/09/19 11:58:00| DNS Socket created at 0.0.0.0, FD 5
>> 2012/09/19 11:58:00| Adding nameserver 80.58.61.250 from squid.conf
>> 2012/09/19 11:58:00| Adding nameserver 8.8.8.8 from squid.conf
>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/10 'ssl_crtd' processes
>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/20 'request_body_max_size.sh'
>> processes
>> 2012/09/19 11:58:00| Logfile: opening log daemon:/var/log/squid3/access.log
>> 2012/09/19 11:58:00| Logfile Daemon: opening log /var/log/squid3/access.log
>> 2012/09/19 11:58:00| Unlinkd pipe opened on FD 31
>> 2012/09/19 11:58:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
>> 2012/09/19 11:58:00| Store logging disabled
>> 2012/09/19 11:58:00| Swap maxSize 15360000 + 262144 KB, estimated 312442 objects
>> 2012/09/19 11:58:00| Target number of buckets: 15622
>> 2012/09/19 11:58:00| Using 16384 Store buckets
>> 2012/09/19 11:58:00| Max Mem size: 262144 KB
>> 2012/09/19 11:58:00| Max Swap size: 15360000 KB
>> 2012/09/19 11:58:00| Rebuilding storage in /mnt/squid/squid3 (clean log)
>> 2012/09/19 11:58:00| Using Least Load store dir selection
>> 2012/09/19 11:58:00| Set Current Directory to /mnt/squid/squid3
>> 2012/09/19 11:58:00| Loaded Icons.
>> 2012/09/19 11:58:00| HTCP Disabled.
>> 2012/09/19 11:58:00| Squid plugin modules loaded: 0
>> 2012/09/19 11:58:00| Adaptation support is off.
>> 2012/09/19 11:58:00| Accepting NAT intercepted HTTP Socket connections at
>> local=0.0.0.0:3128 remote=[::] FD 36 flags=41
>> 2012/09/19 11:58:00| Accepting SSL bumped HTTP Socket connections at
>> local=[::]:3150 remote=[::] FD 37 flags=9
>> 2012/09/19 11:58:00| Store rebuilding is 16.55% complete
>> 2012/09/19 11:58:00| Done reading /mnt/squid/squid3 swaplog (24167 entries)
>> 2012/09/19 11:58:00| Finished rebuilding storage from disk.
>> 2012/09/19 11:58:00| 24167 Entries scanned
>> 2012/09/19 11:58:00| 0 Invalid entries.
>> 2012/09/19 11:58:00| 0 With invalid flags.
>> 2012/09/19 11:58:00| 24167 Objects loaded.
>> 2012/09/19 11:58:00| 0 Objects expired.
>> 2012/09/19 11:58:00| 0 Objects cancelled.
>> 2012/09/19 11:58:00| 0 Duplicate URLs purged.
>> 2012/09/19 11:58:00| 0 Swapfile clashes avoided.
>> 2012/09/19 11:58:00| Took 0.12 seconds (204025.29 objects/sec).
>> 2012/09/19 11:58:00| Beginning Validation Procedure
>> 2012/09/19 11:58:00| Completed Validation Procedure
>> 2012/09/19 11:58:00| Validated 24167 Entries
>> 2012/09/19 11:58:00| store_swap_size = 732468.00 KB
>> 2012/09/19 11:58:01| storeLateRelease: released 0 objects
>> (ssl_crtd): Cannot create ssl certificate or private key.
>> 2012/09/19 12:03:20| WARNING: ssl_crtd #1 exited
>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>> 2012/09/19 12:03:20| Starting new helpers
>> 2012/09/19 12:03:20| helperOpenServers: Starting 1/10 'ssl_crtd' processes
>> 2012/09/19 12:03:20| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper
>> return <NULL> reply
>> (ssl_crtd): Cannot create ssl certificate or private key.
>> 2012/09/19 12:03:20| WARNING: ssl_crtd #2 exited
>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>> 2012/09/19 12:03:20| Closing HTTP port 0.0.0.0:3128
>> 2012/09/19 12:03:20| Closing HTTP port [::]:3150
>> 2012/09/19 12:03:20| storeDirWriteCleanLogs: Starting...
>> 2012/09/19 12:03:20| Finished. Wrote 24195 entries.
>> 2012/09/19 12:03:20| Took 0.02 seconds (1321120.45 entries/sec).
>> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
>>
>> Squid Cache (Version 3.2.1): Terminated abnormally.
>> CPU Usage: 1.896 seconds = 0.740 user + 1.156 sys
>> Maximum Resident Size: 144640 KB
>> Page faults with physical i/o: 0
>> Memory usage for squid via mallinfo():
>> total space in arena: 18900 KB
>> Ordinary blocks: 18674 KB 54 blks
>> Small blocks: 0 KB 1 blks
>> Holding blocks: 37552 KB 9 blks
>> Free Small blocks: 0 KB
>> Free Ordinary blocks: 225 KB
>> Total in use: 56226 KB 297%
>> Total free: 225 KB 1%
>>
>>
>> ------------------------------------------------------------------------------
>> configure and kernel versions:
>>
>> kernel: 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64
>> x86_64 x86_64 GNU/Linux
>>
>> Squid Cache: Version 3.2.1
>> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
>> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
>> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
>> '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode'
>> '--disable-dependency-tracking' '--disable-silent-rules'
>> '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
>> '--mandir=/usr/share/man' '--with-cppunit-basedir=/usr' '--enable-inline'
>> '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd'
>> '--enable-removal-policies=lru,heap' '--enable-delay-pools'
>> '--enable-cache-digests' '--enable-underscores' '--enable-icap-client'
>> '--enable-follow-x-forwarded-for' '--enable-auth-basic' '--enable-auth-digest'
>> '--enable-auth-ntlm' '--enable-auth-negotiate'
>> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
>> '--enable-ntlm-auth-helpers=smb_lm,'
>> '--enable-digest-auth-helpers=ldap,password'
>> '--enable-negotiate-auth-helpers=squid_kerb_auth'
>> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
>> '--enable-arp-acl' '--enable-esi' '--enable-zph-qos' '--enable-wccpv2'
>> '--disable-translation' '--with-logdir=/var/log/squid3'
>> '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536'
>> '--with-large-files' '--with-default-user=proxy' '--enable-ssl'
>> '--enable-ssl-crtd' '--disable-epoll' '--enable-linux-netfilter'
>> 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector
>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>> 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now'
>> 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector
>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>>
>>
>> ------------------------------------------------------------------------------
>> i get this in dmesg:
>>
>> [3433312.743391] init: squid3 main process (29801) terminated with status 1
>> [3433417.181960] ssl_crtd[29848]: segfault at 0 ip 00007f2ee2494ac5 sp
>> 00007fff51dbd260 error 4 in ssl_crtd[7f2ee248b000+10000]
>> [3433417.328898] init: squid3 main process (29847) terminated with status 1
>> [3433557.765878] init: squid3 main process (29877) killed by KILL signal
>> [3433944.030777] init: squid3 main process (30077) terminated with status 1
>>
>>
>> ------------------------------------------------------------------------------
>>
>
> OK, ssl_crtd is dying with a segfault when it is handling a request.
>
> What is in the squid_ssl_db/size file? If it is empty (0 bytes), that would cause a crash in older (pre-3.2.1) squid.
>
> Otherwise, is there an ssl_crtd core file in the squid log directory? A stack trace from it might help diagnose the problem.
>
> Guy
>
Thanks for reply.
i checked the squid_ssl_db/size because i found the empty file problem searching
for my own problem in the mailing list, it's ok in my host, the file have the
content "139264" right now.
I can't found the core file, do i need to do something for it to generate? maybe
a configure script option or squid.conf change to activate it?
Regards,
Miguel Angel.
