Hey David, It will take me more then couple seconds to write an example threaded python helper however it’s pretty simple why this is helper is slow. It uses a select statement and threading in a very wrong way. Before anything else try to compare the right helpers between php and python. The next example helper can be used in comparison to a PHP helper and is much faster: https://gist.githubusercontent.com/elico/03938e3a796c53f7c925872bade78195/raw/85f46ce58db12f30ed99a46c5f300dd8be401674/helper-1.py #!/usr/bin/env python import sys import time #set debug mode for True or False debug = False #debug = True while True: line = sys.stdin.readline().strip() arr = line.split() msg = "" client = "" if debug: sys.stderr.write("__debug info__" + str(time.time()) +": \"" + line + "\"\n") if client and time.time() - time.mktime(time.gmtime(client)) < int(time.time() - logintime*60): sys.stdout.write(arr[0] + " OK \n") if debug: sys.stderr.write("__debug info__ : \"" + line + "\" and time in db is: "+ str(client) +"\n") else: sys.stdout.write(arr[0] + " ERR \n") if debug: sys.stderr.write("__debug info__ : " + line + '\n') sys.stdout.flush() ## END If you have a specific API you want to try and test the requests against let me know and I will try to give an example via this: * HTTP * DNS * Others With the above example you would just need more helpers and add concurrency support to the squid external_acl helper configuration. With enough helpers the stdin buffers will be enough to compensate the missing threads implementation. Eliezer ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: mailto:ngtech1ltd@xxxxxxxxx From: squid-users <squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx> On Behalf Of David Touzeau Sent: Friday, February 4, 2022 16:29 To: squid-users@xxxxxxxxxxxxxxxxxxxxx Subject: Re: external helper development Elizer, Thanks for all this advice and indeed your arguments are valid between opening a socket, sending data, receiving data and closing the socket unlike direct access to a regex or a memory entry even if the calculation has already been done. But what surprises me the most is that we have produced a python plugin in thread which I provide you a code below. The php code is like your mentioned example ( No thread, just a loop and output OK ) Results are after 6k requests, squid freeze and no surf can be made as with PHP code we can up to 10K requests and squid is happy really, we did not understand why python is so low. Here a python code using threads #!/usr/bin/env python import os import sys import time import signal import locale import traceback import threading import select import traceback as tb class ClienThread(): def __init__(self): self._exiting = False self._cache = {} def exit(self): self._exiting = True def stdout(self, lineToSend): try: sys.stdout.write(lineToSend) sys.stdout.flush() except IOError as e: if e.errno==32: # Error Broken PIPE!" pass except: # other execpt pass def run(self): while not self._exiting: if sys.stdin in select.select([sys.stdin], [], [], 0.5)[0]: line = sys.stdin.readline() LenOfline=len(line) if LenOfline==0: self._exiting=True break if line[-1] == '\n':line = line[:-1] channel = None options = line.split() try: if options[0].isdigit(): channel = options.pop(0) except IndexError: self.stdout("0 OK first=ERROR\n") continue # Processing here try: self.stdout("%s OK\n" % channel) except: self.stdout("%s ERROR first=ERROR\n" % channel) class Main(object): def __init__(self): self._threads = [] self._exiting = False self._reload = False self._config = "" for sig, action in ( (signal.SIGINT, self.shutdown), (signal.SIGQUIT, self.shutdown), (signal.SIGTERM, self.shutdown), (signal.SIGHUP, lambda s, f: setattr(self, '_reload', True)), (signal.SIGPIPE, signal.SIG_IGN), ): try: signal.signal(sig, action) except AttributeError: pass def shutdown(self, sig = None, frame = None): self._exiting = True self.stop_threads() def start_threads(self): sThread = ClienThread() t = threading.Thread(target = sThread.run) t.start() self._threads.append((sThread, t)) def stop_threads(self): for p, t in self._threads: p.exit() for p, t in self._threads: t.join(timeout = 1.0) self._threads = [] def run(self): """ main loop """ ret = 0 self.start_threads() return ret if __name__ == '__main__': # set C locale locale.setlocale(locale.LC_ALL, 'C') os.environ['LANG'] = 'C' ret = 0 try: main = Main() ret = main.run() except SystemExit: pass except KeyboardInterrupt: ret = 4 except: sys.exit(ret) Le 04/02/2022 à 07:06, Eliezer Croitoru a écrit : And about the cache of each helpers, the cost of a cache on a single helper is not much in terms of memory comparing to some network access. Again it’s possible to test and verify this on a loaded system to get results. The delay itself can be seen from squid side in the cache manager statistics. You can also try to compare the next ruby helper: https://wiki.squid-cache.org/EliezerCroitoru/SessionHelper About a shared “base” which allows helpers to avoid computation of the query…. It’s a good argument, however it depends what is the cost of pulling from the cache compared to calculating the answer. A very simple string comparison or regex matching would probably be faster than reaching a shared storage in many cases. Also take into account the “concurrency” support from the helper side. A helper that supports parallel processing of requests/lines can do better then many single helpers in more than once use case. In any case I would suggest to enable requests concurrency from squid side since the STDIN buffer will emulate some level of concurrency by itself and will allow squid to keep going forward faster. Just to mention that SquidGuard have used a single helper cache for a very long time, ie every single SquidGuard helper has it’s own copy of the whole configuration and database files in memory. And again, if you do have any option to implement a server service model and that the helpers will contact this main service you will be able to implement much faster internal in-memory cache compared to a redis/memcahe/other external daemon(need to be tested). A good example for this is ufdbguard which has helpers that are clients of the main service which does the whole heavy lifting and also holds one copy of the DB. I have implemented SquidBlocker this way and have seen that it out-performs any other service I have tried until now. _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
