On Tue, Nov 10, 2020 at 7:00 PM Frantisek Zatloukal <fzatlouk@xxxxxxxxxx> wrote:
On Tue, Nov 10, 2020 at 6:11 PM Brendan Early <mymindstorm1@xxxxxxxxx> wrote:There seems to be an issue bug caching, it's showing bugs I closed a few weeks ago.Mhm, I've fixed that. For some reason, the sync of these two packages wasn't happening. I'll analyze the problem more deeply tomorrow. Thanks for the report!
Nevermind, I wouldn't have been able to sleep before diagnosing it :) .
So, without running into our backend details too much: We're using custom counters in Redis to make sure we never queue sync of something twice (or more times, if users share their packages or keep refreshing the page before the data are loaded). Once we queue sync for something, we set a counter for that item to 1 and it can't be queued again until it's synced and counter resets to zero.
However, since we're running on a pretty exhausted and overloaded temporary server, OOM sometimes kills some of our workers or even the redis/database. In these cases, when we lose our tasks queue, but counters stay incremented, we would end up not syncing these items ever (with incremented counters but not in the queue).
To workaround this and further increase reliability, we have implemented a watchdog. It simply goes throughout the cache for any items which weren't synced when they should've been and resets the counter for them. It runs once a day.
However, due to insufficient filtering of non-characters for user inputted FAS names, we have ended up with an escaped quote in our database. That wouldn't be a problem since sqlalchemy is doing it's job to fend off SQLInjections but regex in our watchdog code was crashing on escaped quotes.
Thank you once again for reporting this, we have temporarily hot patched the code on the server and will come up with a proper solution momentarily.
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
