So, I got db-koji02 all setup and doing streaming replication from db-koji01. It has no trouble keeping up. The bad news is, it doesn't solve the backup problem. * If I just do a backup on db-koji02 and don't have hot_standby_feedback on, the backup fails because the primary (01) vacuums or otherwise drops rows that the backup is trying to read. * If I do a backup on db-koji02 with hot_standby_feedback enabled, the backup fails due to a 30 second timeout in streaming delay. ie, 02: "hey, I am backing up these rows, keep them for me until I'm done" 01: "sure, no problem" 02: ...backs up... 30 seconds pass, still backing up... 02: woah, I am 30s behind in replication now due to this lock. <boom> Longer term the problem is really going to need to be solved in koji. There's 2 tables that are gigantic: buildroot_listing (203GB) and tasks (93GB). Those tables need to have lots of things inserting into them, so backing them up will always slow things down. Hopefully they can use partitioning or something and fix this upstream. Possible solutions until they do: 1. I can try and increase max_standby_streaming_delay from 30s to say 90s. This might allow it enough time to copy things with all the locks. 2. We could try breaking the replication, doing the backup on 02 and reconnecting it. This may be difficult to automate and I worry that it might not be consistent if we just disconnect. 3. We could just say that having a hot spare is ok for now and kick the can down the road and not worry about backups. Personally, I am willing to try 1 (will need more +1's for freeze break), I don't really like 2 and 3 seems a bit scary, but I guess it could be ok until after freeze. Thoughts? kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
