On Tue, Nov 10, 2020 at 04:57:20PM +0100, Frantisek Zatloukal wrote: > Hello, > > many of you already heard about Fedora Packager Dashboard [0]. In short, > for those who have not, it's a web application and backend aiming to make > the life of Fedora packagers easier. It combines data from multiple sources > (Pagure, Bugzilla, Bodhi, Koschei,...) relevant to the maintainers of > Fedora packages. > > Tracking all these sites can be time consuming, especially if you maintain > dozens of different packages, so the Dashboard provides everything a > packager might need (or at least what we've thought of) - condensed, > cached, searchable and filterable on one page. > > You can check it out/play with it even if you don't maintain any Fedora > packages, there is no authentication needed, just enter any packager’s > username. Feature-wise, it's pretty robust already, and there are more > things to come (like allowing users to authenticate and see private bugs > and more), but the original planned feature set is complete. > > Currently, it's processing only publicly available data. When we'll start > implementing the ability to authenticate and process any private data, > we'll work closely with the Infra team to make sure there are no open holes. > > Since the announcement of the testing phase, it has been running on a > temporary server which is barely keeping up, so I'd like to open up a > conversation about migration to the Infra OpenShift cluster. > > Here is a brief overview of it's internals (I can elaborate more if anybody > needs me to): > > Backend is a Flask/Python application leveraging Celery for planning and > executing the cache refreshes. The API is striving to be as non-blocking as > possible, using asynchronous-inspired behaviour. The client is given the > data currently available in cache, and advised about the completeness > (complete/partial cache misses) via HTTP status code. > > The parameters for cache refreshes can be customized, depending on the > resources we have/can get. Currently, it's retrieving data for most of the > items every 2 hours (with exceptions like package versions which run daily > and are terribly slow). Backend is caching data for PRs, bugs, and > pre-calculated updates/overrides data for users visiting the app at least > once in two weeks. The main storage is a PostgreSQL database, and > optionally, if RAM is not an issue, we have a local memory-cached layer > that can be enabled (we find it not necessary ATM). > > Apart from storing the pre-crunched information from public sources, we > keep timestamps of the last visit for each user. > > Frontend is a React app fetching and displaying data from the backend > (really nothing to add here :) ). > > Based on the OpenShift testing, I've come to the following schema of pods: > > - > > Redis pod (Celery backend) Redis is at its heart a caching system that can be tweaked to behave like a queueing system. If you're using celery, it may be worth checking if you can migrate to amqp (and thus rabbitmq) which should be supported by celery just fine. That would give you the possibility to use the rabbitmq cluster we already have for fedora-messaging. Now, since you're saying in your reply about the caching issue that you're using a custom counter in redis, I don't know if that change would be as trivial as just changing the URL in the celery configuration. > PostgreSQL pod (cache and watchdog data storage) Kevin has already covered this one and I won't comment any further ;-) Thanks for working on this! Pierre _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
