On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote: > Hello, > > is it a known problem that koji notifications arrive way too late? For yes. > example, I got mame build notifications in the wee hours today (31.12) > wheras the builds completed around noon on the 29th: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 > This does not seem normal. Is anybody seeing this too? Everyone. :( The problem here is our notification system ( fmn ) has a number of issues: * The current version is still python2 and crashes from time to time for difficult to debug reasons. So, sometimes it has crashed and takes a bit to realize it's not processing and get restarted. :( We do have a python3 port almost ready to swap in. Once thats in at least we can hopefully debug any crashes. I hope that will happen in the first quarter. * The current version has issues processing when 'large' amounts of messages flood the bus. Examples of this include: - When a beta or final release is 'go' releng tags all the packages to record they were part of the release, so ~23k messages in a few minutes. - when copr rebuilds all rubygems - mass rebuilds, thats ~23*N messages - Most recently, when epel9 branching opened, because say 200 packages branched quicky and merged every commit from rawhide into the epel9 branch and each commit is a message. So, anyhow, I hope the python3 version will help out some, but really we need to do a re-write of the application to better handle lots of messages. The current app is... difficult to use, but super flexible. If we made it easier to use, simpliler, but some less flexable it could help processing speed. There has also been talk about seeing if we can just use rabbitmq for the heavy lifting here (ie, have queues for people who customize messages with those topics and have rabbitmq sort them for us), that might work or help. I hope we can work on this this year. kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
