On Wed, May 17, 2017 at 03:38:25PM -0600, Kevin Fenzi wrote: > On 05/17/2017 03:27 PM, Jeremy Cline wrote: > > Hello everyone, > > > > People have noticed that they aren't getting some FMN notifications, so > > I spent the afternoon tracking down the issue. It looks as though the > > redis instance on notifs-backend01.phx2.fedoraproject.org is configured > > to persist to disk (this is apparently the default config, which is > > surprising to me). In addition, dogpile.cache is apparently not setting > > a TTL on keys despite being configured to do so. > > > > All this means that everything in the FMN redis cache is _really_ stale > > and contains invalid objects. For example, the packages a user has ACLs > > on is cached as a set currently, but the latest version of FMN expects > > a dictionary. This leads to messages not getting sent that should have > > been sent. > > > > I would like to delete the persisted database (/var/lib/redis/dump.rdb) > > and restart redis to wipe the cache. FMN does not like redis going down > > so the workers and backend will likely need to be restarted, and because > > of a different caching issue[0] this will likely take around an hour > > (messages won't be lost, they'll just build up in the rabbitmq queue). > > > > [0] https://github.com/fedora-infra/fmn/pull/190 > > > > +1s? > > +1 here. Do it. +1 for me to Pierre
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
