Ok - I will try to backport it from that patch into the v5 tree I've downloaded today. As we were using the mentioned build I came across these new assertions: 2021/11/16 10:29:46 kid1| assertion failed: StoreMap.cc:241: "anchorAt(anchorId).reading()" 2021/11/16 11:32:51 kid2| assertion failed: Transients.cc:221: "old == e" 2021/11/16 13:02:09 kid2| assertion failed: Transients.cc:221: "old == e" 2021/11/16 13:52:05 kid2| assertion failed: Transients.cc:221: "old == e" 2021/11/16 14:29:41 kid2| assertion failed: store.cc:1108: "store_status == STORE_PENDING" 2021/11/16 15:26:15 kid1| assertion failed: Transients.cc:221: "old == e" 2021/11/16 17:40:21 kid1| assertion failed: cbdata.cc:372: "c->locks > 0" 2021/11/16 17:40:44 kid1| assertion failed: cbdata.cc:115: "cookie == ((long)this ^ Cookie)" (no config changes) My 1w cache.log is about 300MB - without elevated debug options (debug options ALL,1) - so it’s not easy to find something relevant with "9" options enabled... LL -----Original Message----- From: Alex Rousskov [mailto:rousskov@xxxxxxxxxxxxxxxxxxxxxxx] Sent: Tuesday, November 16, 2021 3:42 PM To: Loučanský Lukáš; Squid Users Subject: Re: Too many ERROR: Collapsed forwarding queue overflow for kid2 at 1024 items On 11/16/21 4:38 AM, Loučanský Lukáš wrote: > is it going to be patched only in the v6 version? I hope the existing fix applies to v5 cleanly, and I am ready to help with backporting if it does not. Beyond that, it is in the maintainer hands. I cannot predict whether or when the fix will be officially merged into v5 because I do not understand how those decisions are made. > Anyway - in the morning I run debug with 20,9 to see: > ... > 2021/11/16 09:02:06.496 kid2| assertion failed: Transients.cc:221: "old == e" Unfortunately, I cannot see the cause of the assertion in this short/partial trace -- the problematic actions happened before the trace or were not logged during the trace. Patching your Squid with commit 5210df4 is the best next step IMO. If that patch does not help, then there are probably other bugs that we need to fix in v5 (at least). HTH, Alex. > -----Original Message----- > From: Alex Rousskov [mailto:rousskov@xxxxxxxxxxxxxxxxxxxxxxx] > Sent: Monday, November 15, 2021 5:17 PM > To: Squid Users > Cc: Loučanský Lukáš > Subject: Re: Too many ERROR: Collapsed forwarding queue > overflow for kid2 at 1024 items > > On 11/15/21 7:43 AM, Loučanský Lukáš wrote: > >> 2021/11/14 10:13:30 kid2| assertion failed: Transients.cc:221: "old == e" >> 2021/11/15 08:37:36 kid2| assertion failed: Transients.cc:221: "old == e" >> 2021/11/15 11:54:14 kid1| assertion failed: Transients.cc:221: "old == e" >> 2021/11/15 12:16:27 kid1| assertion failed: Transients.cc:221: "old == e" > > I recommend ignoring queue overflows until the above assertions are fixed because worker deaths cause queue overflows. Your Squid is buggy, and those bugs essentially cause queue overflows. > > The assertion itself is known as Bug 5134: > https://bugs.squid-cache.org/show_bug.cgi?id=5134 > > That bug has a speculative fix (master/v6 commit 5210df4). Please try it if you can. > > > HTH, > > Alex. > > >> -----Original Message----- >> From: Alex Rousskov [mailto:rousskov@xxxxxxxxxxxxxxxxxxxxxxx] >> Sent: Friday, November 12, 2021 5:24 PM >> To: Loučanský Lukáš; squid-users@xxxxxxxxxxxxxxxxxxxxx >> Subject: Re: Too many ERROR: Collapsed forwarding queue >> overflow for kid2 at 1024 items >> >> On 11/11/21 10:19 AM, Loučanský Lukáš wrote: >> >>> recently I'm facing too many ERROR: Collapsed forwarding queue >>> overflow for kid2 at 1024 items lines in my Squid 5.2 log files. >> >> We see those overflows when kids die. Do you see any FATAL messages, assertions, or similar deadly errors in cache.log? >> >> >>> Could someone elaborate how the queue is filled - what is clogging it? >> >> The sender/writer sends messages faster than the recipient/reader is >> reading them, eventually exceeding the queue capacity (i.e. 1024 >> messages). These messages are about Store entries that may need >> synchronization across workers. Each message is very sm > all. >> >> >>> I don't mind too much if I have to turn collapsed forwarding off >> >> Most likely, the problem is not tied to collapsed forwarding. These >> queues were used for collapsed forwarding when they were added, but >> they are used for regular traffic as well in modern SMP Squids. We >> need to change the queue names (and related code/m > essage text) to reflect the expanded nature of these queues. >> >> >> HTH, >> >> Alex. >> _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
