Make broken database start

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi everyone,

(sorry if this is a dup - I'm not sure if my previous message get through)

I need an advice in database recovery. Postgresql 9.2.9.
Last night we moved one of our databases to another server following this procedure: https://wiki.postgresql.org/wiki/Streaming_Replication
Replication, slave becomes master, pgbouncer restarts with new config, everyone happy.
Until soon we noticed strange errors. Turns out data was broken somewhere - we had unique indexes with duplicate records, we had missing chunks in toast tables, and the like.
Scale of the problems wasn't clear and I fixed some of the most obvious problems - recreated indices, cleared duplicates. Around 0:30 I started database-wide VACUUM, it started printing messages about fixing things. Everything seemed to work normally.
Around 3:30 database stopped responding to queries; it was noticed around 9:30 and I attempted to stop VACUUM (ctrl+c in psql command line). 30 minutes and nothing. I decided to restart postmaster. After another 30 minutes we switched back to old server. Already more than 6 hours passed and new server postmaster is still shutting down. And I would like to get it back to restore yesterday data.

At this point I could do with new server anything that will get database up and running sooner. kill -9, start, wait for recovery? Server reboot?

Any help will be greatly appreciated.

Best regards,
Dmitriy Shalashov

[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux