On Thu, Nov 12, 2020 at 10:03 AM Pierre-Yves Chibon <pingou@xxxxxxxxxxxx> wrote:
Redis is at its heart a caching system that can be tweaked to behave like a
queueing system. If you're using celery, it may be worth checking if you can
migrate to amqp (and thus rabbitmq) which should be supported by celery just
fine.
That would give you the possibility to use the rabbitmq cluster we already have
for fedora-messaging.
Now, since you're saying in your reply about the caching issue that you're using
a custom counter in redis, I don't know if that change would be as trivial as
just changing the URL in the celery configuration.
Hmm, we've looked at it a bit with jskladan. The change is definitely doable but not trivial. We're making use of our redis counter being thread-safe and atomic.
The compromise might be switching celery backend to rabbitmq and leaving the counter to be stored in redis and trying to change it after the deployment? But that'd still mean we'll have to run Redis somewhere :( .
Also, just a heads up notice, we're queuing roughly 20-30 thousand celery tasks every 2 hours (depending on set sync interval for oraculum cache). That might get higher once it's marketed in more places. I think rabbitmq should be able to handle that, but I don't know how many resources there are on the Fedora rabbitmq server.
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
