On Thu, Nov 12, 2020 at 06:03:01PM +0100, Frantisek Zatloukal wrote: > On Thu, Nov 12, 2020 at 10:03 AM Pierre-Yves Chibon <pingou@xxxxxxxxxxxx> > wrote: > > > Redis is at its heart a caching system that can be tweaked to behave like a > > queueing system. If you're using celery, it may be worth checking if you > > can > > migrate to amqp (and thus rabbitmq) which should be supported by celery > > just > > fine. > > That would give you the possibility to use the rabbitmq cluster we already > > have > > for fedora-messaging. > > > > Now, since you're saying in your reply about the caching issue that you're > > using > > a custom counter in redis, I don't know if that change would be as trivial > > as > > just changing the URL in the celery configuration. > > > > Hmm, we've looked at it a bit with jskladan. The change > is definitely doable but not trivial. We're making use of our redis counter > being thread-safe and atomic. > > The compromise might be switching celery backend to rabbitmq and leaving > the counter to be stored in redis and trying to change it after the > deployment? But that'd still mean we'll have to run Redis somewhere :( . > > Also, just a heads up notice, we're queuing roughly 20-30 thousand celery > tasks every 2 hours (depending on set sync interval for oraculum cache). > That might get higher once it's marketed in more places. I think rabbitmq > should be able to handle that, but I don't know how many resources there > are on the Fedora rabbitmq server. We have a 3 node cluster that does a pretty good job. I think it should be able to handle this, but we could test in stg in any case. kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
