You were right...
As soon as i add to my script the check for empty line squid can now
shutdown the redirector childrens...
==
if ( not line ):
EOF = True
continue
==
About unbuffered i/o, well i was understanding that we only need the
stdout to be unbuffered that why i was always flushing the stdout but
well you seems to got mucho more experience in this field so i was
just follow you advice ;)
Offtopic questions:
1) I see that squid is sending \n (0x0a) to the redirects while not
doing any job, should i response those querys with another \n (0x0a)
or just pass it until i see a good argument line.
2) Any advice on achieving concurrency in python?
Thanks for the help
Best Regards
El día 7 de septiembre de 2010 11:06, Diego Woitasen
<diegows@xxxxxxxxxxxx> escribió:
> 2010/9/7 Jorge Iván Burgos Aguilar <jorgeburgos@xxxxxxxxxxxxxxxxxxx>:
>> Hi,
>>
>> I have been building for the past week a special redirector for squid
>> in python 2.6.x; the redirectors works good they do their job but when
>> i hit shutdown or reload (squid -k shutdown|reload) they just won't
>> simple die and in fact they start to became huge resource monster and
>> lag the cpu until obviously i kill them manually (which is in fact the
>> workaround i have just made for when squid hit reload or shutdown on
>> the init.d scripts).
>> I don't know if its a bug or configuration problem so here are some
>> relevant logs...
>>
>> ==cache.log==
>> 2010/09/06 13:55:04.217| helperOpenServers: Starting 2/2 'python26' processes
>> .....
>> 2010/09/06 13:55:41.808| helperShutdown: redirector #1 shutting down.
>> 2010/09/06 13:55:41.808| comm_close: start closing FD 9
>> 2010/09/06 13:55:41.808| The AsyncCall comm_close_start constructed,
>> this=0x2b7fce5b3590 [call66]
>> 2010/09/06 13:55:41.808| comm.cc(1611) will call comm_close_start(FD 9) [call66]
>> 2010/09/06 13:55:41.808| comm.cc(1195) commSetTimeout: FD 9 timeout -1
>> 2010/09/06 13:55:41.808| comm.cc(1206) commSetTimeout: FD 9 timeout -1
>> 2010/09/06 13:55:41.808| commio_finish_callback: called for FD 9 (-10, 11)
>> 2010/09/06 13:55:41.808| comm.cc(165) will call SomeCommReadHandler(FD
>> 9, errno=11, flag=-10, data=0x2b7fce58b528, size=0,
>> buf=0x2b7fce58b5f0) [call4]
>> 2010/09/06 13:55:41.808| commCallCloseHandlers: FD 9
>> 2010/09/06 13:55:41.808| commCallCloseHandlers: ch->handler=0x2b7fce568a80*1
>> 2010/09/06 13:55:41.808| comm.cc(1460) will call SomeCloseHandler(FD
>> 9, data=0x2b7fce58b528) [call3]
>> 2010/09/06 13:55:41.809| The AsyncCall comm_close_complete
>> constructed, this=0x2b7fce5b3600 [call67]
>> 2010/09/06 13:55:41.809| comm.cc(1643) will call
>> comm_close_complete(FD 9) [call67]
>> 2010/09/06 13:55:41.809| helperShutdown: redirector #2 shutting down.
>> 2010/09/06 13:55:41.809| comm_close: start closing FD 11
>> 2010/09/06 13:55:41.809| The AsyncCall comm_close_start constructed,
>> this=0x2b7fce5b3670 [call68]
>> 2010/09/06 13:55:41.809| comm.cc(1611) will call comm_close_start(FD
>> 11) [call68]
>> 2010/09/06 13:55:41.809| comm.cc(1195) commSetTimeout: FD 11 timeout -1
>> 2010/09/06 13:55:41.809| comm.cc(1206) commSetTimeout: FD 11 timeout -1
>> 2010/09/06 13:55:41.809| commio_finish_callback: called for FD 11 (-10, 11)
>> 2010/09/06 13:55:41.809| comm.cc(165) will call SomeCommReadHandler(FD
>> 11, errno=11, flag=-10, data=0x2b7fce58d7e8, size=0,
>> buf=0x2b7fce58d8b0) [call8]
>> 2010/09/06 13:55:41.809| commCallCloseHandlers: FD 11
>> 2010/09/06 13:55:41.809| commCallCloseHandlers: ch->handler=0x2b7fce58f8c0*1
>> 2010/09/06 13:55:41.809| comm.cc(1460) will call SomeCloseHandler(FD
>> 11, data=0x2b7fce58d7e8) [call7]
>> 2010/09/06 13:55:41.809| The AsyncCall comm_close_complete
>> constructed, this=0x2b7fce58ad70 [call69]
>> 2010/09/06 13:55:41.809| comm.cc(1643) will call
>> comm_close_complete(FD 11) [call69]
>> .....
>> 2010/09/06 13:55:41.810| Open FD UNSTARTED 9 python26 #1
>> 2010/09/06 13:55:41.810| Open FD UNSTARTED 11 python26 #2
>> ==cache.log==
>>
>> And a sample/stub of the redirector python script if it helps...
>> ==python redirector script==
>> #!/usr/bin/python26
>> import sys
>>
>> # Variables
>> EOF = False
>> RecentBorn = True
>>
>> # Life Cycle
>> try:
>> while ( not EOF ):
>> if ( RecentBorn ):
>> RecentBorn = False
>> else:
>> sys.stdout.flush()
>>
>> line = sys.stdin.readline()
>> sys.stdout.write("\n")
>>
>> except KeyboardInterrupt:
>> EOF = True
>> ==python redirector script==
>>
>> And my running box of course:
>> # uname -msr
>> Linux 2.6.18-194.11.3.el5.centos.plus x86_64
>> # squid -v
>> Squid Cache: Version 3.1.8
>> configure options: '--build=x86_64-redhat-linux-gnu'
>> '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu'
>> '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
>> '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
>> '--datadir=/usr/share' '--includedir=/usr/include'
>> '--libdir=/usr/lib64' '--libexecdir=/usr/libexec'
>> '--sharedstatedir=/usr/com' '--mandir=/usr/share/man'
>> '--infodir=/usr/share/info' '--exec_prefix=/usr'
>> '--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
>> '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
>> '--with-logdir=$(localstatedir)/log/squid'
>> '--with-pidfile=$(localstatedir)/run/squid.pid'
>> '--disable-dependency-tracking' '--enable-arp-acl'
>> '--enable-follow-x-forwarded-for'
>> '--enable-auth=basic,digest,ntlm,negotiate'
>> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth'
>> '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth'
>> '--enable-digest-auth-helpers=password,ldap,eDirectory'
>> '--enable-negotiate-auth-helpers=squid_kerb_auth'
>> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
>> '--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
>> '--enable-delay-pools' '--enable-epoll' '--enable-icap-client'
>> '--enable-ident-lookups' '--enable-linux-netfilter'
>> '--enable-referer-log' '--enable-removal-policies=heap,lru'
>> '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs'
>> '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio'
>> '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
>> '--with-openssl' '--with-pthreads' '--disable-loadable-modules'
>> 'build_alias=x86_64-redhat-linux-gnu'
>> 'host_alias=x86_64-redhat-linux-gnu'
>> 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
>> -fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
>> 'LDFLAGS=-pie' 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall
>> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
>> --param=ssp-buffer-size=4 -m64 -mtune=generic'
>> --with-squid=/home/constructor/rpmbuild/BUILD/squid-3.1.8
>> --enable-ltdl-convenience
>>
>> Best Regards
>>
>
> The redirectors and external_acls must finish when they receive an
> empty line. The skeleton of your redirector should be something like
> this:
>
> #!/usr/bin/python -u <--- -u is very important, it disables
> buffering on stdin, stdout and stderr
>
> import sys
>
> while True:
> line = sys.stdin.readline()
> if not line:
> break
>
> do_something(line)
>
>
> Regards,
> DIego
>
> --
> Diego Woitasen
> XTECH
>
