|
On 10/11/2016 09:07 PM, Kevin Fenzi
wrote:
Kevin,Greetings. After a bunch of playing around, I managed to get bidirectional database replication working for koji.stg.fedoraproject.org. Basic outline: db-koji01.stg and db-koji02.stg both have postgresql-9.4-bdr installed. init a koji db on both. create a koji user and set password on both. restore prod db dump on one of them. enable replication of the koji db. wait for a while for them to sync up (It took about 4.5 hours to sync ~170GB of database) run the staging sql script on either node. I then added keepalived to keep a 'application ip' between the two of them (favoring 01). Then I pointed koji01.stg at the application ip (well, I really just removed it's /etc/hosts entry for db-koji) I tested switching the ip back and forth. I tested rebooting one node or the other. I tested disabling keepalived on one and rebooting (so the other one became and stayed primary). Everything seemed to work. ;) So, next steps: 1. Pound on koji.stg and see if anything breaks. ;) I tried to enable koschei, but I think we have it currently set too not do staging builds. If we could enable it that might be good. If anyone notices anything broken on koji.stg, please let me know. If you can think of some common cases we should test, please let me know that too. ;) 2. I am going to look at setting up another pair and get this all in ansible and then see about migrating other staging services. I'm hopeful that if koji works all our less database heavy apps will work ok too. 3. If everything keeps looking good, move to production. Longer term what does this mean? It means we can do updates/reboot cycles with pretty close to no downtime. We may have to be clever about the openvpn hub, but if we can reboot database servers as desired we may be able to avoid maint windows entirely, or at least reduce them a great deal. kevin Any tests in particular an apprentice could pound on it to check ? Not honestly sure I have access to koji.stg but I'd be game for testing common and edge cases fas:linuxmodder -- Corey Sheldon Freelance IT Consultant, FamNa, Hacktivist +1.310.909.7672 +1.310.737.7237 (Text only) sheldon.corey@xxxxxxxxxxxxxxx https://gist.github.com/linux-modder/ac5dc6fa211315c633c9 PGP Keys: 0x8B4E89435A88E539 | C006 564F FA67 CDEA E29B F202 8B4E 8943 5A88 E539 0x59276298d2264944 | 6292 9ABD 6374 6AA7 6D4B 730F 5927 6298 D226 4944 Disclaimer: ALL Correspondence shall be deemed as private and confidential, re-distrubution is discouraged unless requested in the the correspondence in question. |
Attachment:
0x5A88E539.asc
Description: application/pgp-keys
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
