On Sun, Jan 02, 2022 at 12:03:25PM -0800, Kevin Fenzi wrote:
On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote:
Hello,
is it a known problem that koji notifications arrive way too late? For
yes.
example, I got mame build notifications in the wee hours today (31.12)
wheras the builds completed around noon on the 29th:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260
https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254
https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253
This does not seem normal. Is anybody seeing this too?
Everyone. :(
The problem here is our notification system ( fmn ) has a number of
issues:
* The current version is still python2 and crashes from time to time for
difficult to debug reasons. So, sometimes it has crashed and takes a bit
to realize it's not processing and get restarted. :( We do have a
python3 port almost ready to swap in. Once thats in at least we can
hopefully debug any crashes. I hope that will happen in the first
quarter.
* The current version has issues processing when 'large' amounts of
messages flood the bus. Examples of this include:
- When a beta or final release is 'go' releng tags all the packages to
record they were part of the release, so ~23k messages in a few
minutes.
- when copr rebuilds all rubygems
- mass rebuilds, thats ~23*N messages
- Most recently, when epel9 branching opened, because say 200 packages
branched quicky and merged every commit from rawhide into the epel9
branch and each commit is a message.
I'm right now the unlucky receiver of this. It's really annoying. Facter
has a ~15 year history so it's a lot of commits.
Couldn't this be grouped? I don't see a multi commit viewer in Pagure
where you can say $start..$end (like regular git, github, gitlab and
others). That would save a lot of notifications and actually be
effective.
So, anyhow, I hope the python3 version will help out some, but really we
need to do a re-write of the application to better handle lots of
messages. The current app is... difficult to use, but super flexible.
If we made it easier to use, simpliler, but some less flexable it could
help processing speed. There has also been talk about seeing if we can
just use rabbitmq for the heavy lifting here (ie, have queues for people
who customize messages with those topics and have rabbitmq sort them for
us), that might work or help. I hope we can work on this this year.
Is there anything people could do to help here? Like is there a roadmap?
I think the source is https://github.com/fedora-infra/fmn and there is
https://github.com/fedora-infra/fmn/issues/143 but that has very little
activity.
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
