+1 on #1
-re
On 2019-10-22 16:56, Stephen John Smoogen wrote:
On Tue, 22 Oct 2019 at 16:55, Mohan Boddu <mboddu@xxxxxxxxxx> wrote:
On Mon, Oct 21, 2019 at 2:48 PM Kevin Fenzi <kevin@xxxxxxxxx> wrote:
So, I got db-koji02 all setup and doing streaming replication from
db-koji01. It has no trouble keeping up.
The bad news is, it doesn't solve the backup problem.
* If I just do a backup on db-koji02 and don't have hot_standby_feedback
on, the backup fails because the primary (01) vacuums or otherwise drops
rows that the backup is trying to read.
* If I do a backup on db-koji02 with hot_standby_feedback enabled, the
backup fails due to a 30 second timeout in streaming delay. ie,
02: "hey, I am backing up these rows, keep them for me until I'm done"
01: "sure, no problem"
02: ...backs up... 30 seconds pass, still backing up...
02: woah, I am 30s behind in replication now due to this lock. <boom>
Longer term the problem is really going to need to be solved in koji.
There's 2 tables that are gigantic: buildroot_listing (203GB) and tasks
(93GB). Those tables need to have lots of things inserting into them, so
backing them up will always slow things down. Hopefully they can use
partitioning or something and fix this upstream.
Possible solutions until they do:
1. I can try and increase max_standby_streaming_delay from 30s to say
90s. This might allow it enough time to copy things with all the locks.
+1
2. We could try breaking the replication, doing the backup on 02 and
reconnecting it. This may be difficult to automate and I worry that it
might not be consistent if we just disconnect.
3. We could just say that having a hot spare is ok for now and kick the
can down the road and not worry about backups.
Personally, I am willing to try 1 (will need more +1's for freeze
break), I don't really like 2 and 3 seems a bit scary, but I guess it
could be ok until after freeze.
Thoughts?
I prefer #1 compared to #2 and #3.
I think for the next week while Kevin is on vacation.. #1 is what we
should go with.
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
