On Wed, Oct 16, 2019 at 11:01:48PM -0400, Stephen John Smoogen wrote: > On Wed, 16 Oct 2019 at 18:35, Kevin Fenzi <kevin@xxxxxxxxx> wrote: > > > > Greetings folks. > > > > As you may know, currently database backups on db-koji01 are causing > > very heavy load, disrupting our users builds ( > > https://pagure.io/fedora-infrastructure/issue/8292 ) > > so, they are currently disabled. > > > > However, not having current backups is not a good thing, IMHO. > > > > So, I am considering the idea of adding a db-koji02 vm (also rhel7 using > > the same postgres version that db-koji01 is) and enabling streaming > > replication from db-koji01 -> db-koji02 and then once thats working, run > > the database backups on db-koji02. > > > > It turns out this doesn't require that many changes on db-koji01: > > * adding a replication user > > * Setting 3 new lines of postgresql config and restarting: > > wal_level = 'hot_standby' > > max_wal_senders = 10 > > wal_keep_segments = 100 > > (May need to adjust senders and keep segments) > > > > All the other changes are on db-koji02: > > * create/setup the vm > > * run pg_basebackup to pull all the current data from 01 > > * setup postgresql.conf and recovery.conf files > > * start server and confirm it keeps up with 01 > > * run pg_dump and confirm it keeps up with 01 > > > > This is, of course a really big change in a freeze to a criticial > > service, so I'd like to get thoughts from others about it. > > > > Should we wait until after freeze and do without backups until then > > (note, that we have never had to restore this db from backups in the > > past, although we have dumped/restored it to move to newer postgres > > versions)? > > > > Is there something else we can do thats easier to mitigate the issues > > right now? > > > > Thoughts? ideas? Rotten fruit? > > I think this would be a good thing to do. I guess we can put this on > virthost04 once we get it to actually boot versus hang on tang? Well, it needs to be a bvirthost, so I was going to use bvirthost12. It's pretty lightly loaded and has lots of memory free. kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
