Greetings. After a bunch of playing around, I managed to get bidirectional database replication working for koji.stg.fedoraproject.org. Basic outline: db-koji01.stg and db-koji02.stg both have postgresql-9.4-bdr installed. init a koji db on both. create a koji user and set password on both. restore prod db dump on one of them. enable replication of the koji db. wait for a while for them to sync up (It took about 4.5 hours to sync ~170GB of database) run the staging sql script on either node. I then added keepalived to keep a 'application ip' between the two of them (favoring 01). Then I pointed koji01.stg at the application ip (well, I really just removed it's /etc/hosts entry for db-koji) I tested switching the ip back and forth. I tested rebooting one node or the other. I tested disabling keepalived on one and rebooting (so the other one became and stayed primary). Everything seemed to work. ;) So, next steps: 1. Pound on koji.stg and see if anything breaks. ;) I tried to enable koschei, but I think we have it currently set too not do staging builds. If we could enable it that might be good. If anyone notices anything broken on koji.stg, please let me know. If you can think of some common cases we should test, please let me know that too. ;) 2. I am going to look at setting up another pair and get this all in ansible and then see about migrating other staging services. I'm hopeful that if koji works all our less database heavy apps will work ok too. 3. If everything keeps looking good, move to production. Longer term what does this mean? It means we can do updates/reboot cycles with pretty close to no downtime. We may have to be clever about the openvpn hub, but if we can reboot database servers as desired we may be able to avoid maint windows entirely, or at least reduce them a great deal. kevin
Attachment:
pgpDNBYLQCx1b.pgp
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
