On Mon, Oct 21, 2019 at 2:48 PM Kevin Fenzi <kevin@xxxxxxxxx> wrote: > > So, I got db-koji02 all setup and doing streaming replication from > db-koji01. It has no trouble keeping up. > > The bad news is, it doesn't solve the backup problem. > > * If I just do a backup on db-koji02 and don't have hot_standby_feedback > on, the backup fails because the primary (01) vacuums or otherwise drops > rows that the backup is trying to read. > > * If I do a backup on db-koji02 with hot_standby_feedback enabled, the > backup fails due to a 30 second timeout in streaming delay. ie, > 02: "hey, I am backing up these rows, keep them for me until I'm done" > 01: "sure, no problem" > 02: ...backs up... 30 seconds pass, still backing up... > 02: woah, I am 30s behind in replication now due to this lock. <boom> > > Longer term the problem is really going to need to be solved in koji. > There's 2 tables that are gigantic: buildroot_listing (203GB) and tasks > (93GB). Those tables need to have lots of things inserting into them, so > backing them up will always slow things down. Hopefully they can use > partitioning or something and fix this upstream. > > Possible solutions until they do: > > 1. I can try and increase max_standby_streaming_delay from 30s to say > 90s. This might allow it enough time to copy things with all the locks. +1 > > 2. We could try breaking the replication, doing the backup on 02 and > reconnecting it. This may be difficult to automate and I worry that it > might not be consistent if we just disconnect. > > 3. We could just say that having a hot spare is ok for now and kick the > can down the road and not worry about backups. > > Personally, I am willing to try 1 (will need more +1's for freeze > break), I don't really like 2 and 3 seems a bit scary, but I guess it > could be ok until after freeze. > > Thoughts? I prefer #1 compared to #2 and #3. > > kevin > _______________________________________________ > infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
