On 15/10/2020 14:59, Alex Rousskov wrote: > On 10/15/20 4:07 AM, Stephen Borrill wrote: >> At a few installations of squid 4.12 (patched for GREASE) on NetBSD >> 9, I'm seeing that occasionally one of the listening ports no longer >> accepts connections (it doesn't reject them, but a connection does >> not get established).> The port appears random; it's not the same >> every time and isn't related to ports with SSL interception. A >> restart of squid fixes it. > >> Looking through the logs, this appears to coincide with lines such as: >> >> 2020/10/14 22:32:16 kid1| ERROR: getsockname() failed to locate local-IP >> on local=[::] remote=10.0.106.147:61996 FD 25 flags=1: (22) Invalid argument >> 2020/10/14 22:32:16 kid1| BUG 3556: FD 25 is not an open socket. >> >> This looks similar to Alex Rousskov's recent observations: >> https://bugs.squid-cache.org/show_bug.cgi?id=3556#c15 > > Please keep in mind that those "BUG 3556" messages warn us about Squid > bugs elsewhere/somewhere in Squid code. For each particular message > instance, the exact bug is unknown a priori, and several different bugs > have triggered these messages in the past. While the original bug 3556 > report was for a specific bug, the log messages were not (and are not). > > >> However, we have also seen with at sites where there is no SSL >> interception (the above lines are from such an installation). > >> 1) Am I right that triggering BUG 3556 could lead to the described symptoms? > > I would rephrase this as "Failure to obtain the (intended) IP address of > an (intercepted) connection leads to BUG 3556 messages." > > >> 2) Is this a squid bug or triggered by a problem in the underlying OS? > > Those "BUG 3556" messages indicate a Squid bug. There is no question > about that. However, the ERROR messages may indicate a Squid > bug/deficiency and/or an environment (OS configuration, etc.) problem. > In summary, you are dealing with multiple problems here. You should > focus on the ERROR messages, not "BUG 3556" messages. > > >> If the latter, where to start looking? > > Check system log for errors. Perhaps you are exhausting some system > resource? That's always my first port of call anyway, there are no reports. > I would also try to map ERROR messages to client transactions in hope to > spot some common pattern behind those failed transactions. > Unfortunately, I do not know whether Squid (especially Squid v4) would > log these failed transactions. > > >> 3) What workarounds are there? > > a) Monitor logs and automatically restart the Squid instance if needed. > > b) Patch Squid to kill the affected process. Adding "assert(false);" > after the ERROR message is printed in Comm::TcpAcceptor::oldAccept() > will kill the process. Killing a single worker may or may not be enough > in SMP mode; it would be interesting and potentially useful to know > whether that is enough. We aren't using SMP mode. > You may be able to easily test your workaround using the trick I > outlined in https://bugs.squid-cache.org/show_bug.cgi?id=3556#c15 I have also been pointed to your comment here: https://bugs.squid-cache.org/show_bug.cgi?id=5069#c1 I'm currently testing the COMM_ERROR -> NOMESSAGE patch as bug 5069 sounds identical. >> Given that a restart fixes it, in some >> respects it would be better for squid to quit so it can be restarted >> rather than continue to run in a half-working state. > > Yes, earlier Squids were written using the "Do whatever you can to stay > up" or "Damn the torpedoes!" principle. FWIW, I am pushing for reversing > the relevant code logic to follow the "Squid instance that cannot > provide an essential service explicitly requested by the admin should > quit with an error" principle, but it will take time to achieve that ideal. > > >> 4) Related to 3), are there any other ways to detect the problem when it >> is happening besides parsing logs or testing if all ports are accepting >> connections? This could be used to trigger an automated restart as a >> temporary workaround. > > Yes, see suggestion 3b above. > > > HTH, > > Alex. > _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
