Thank you for the feedback, Alvaro.
Unfortunately, the database is no longer "dumpable". We were able to do
a pg_dump yesterday morning (12 hours after the crash + purging the
pg_clog) but if we try one now, we get the following error:
unexpected chunk number 1 (expected 0) for toast value 8282331 in
pg_toast_38651
Looking at our data, there seem to be 6 tables that have corrupt
records. Doing a SELECT * for one of those records, will return a
similar error:
missing chunk number 0 for toast value 8288522 in pg_toast_5572299
What is the best way to go from here? Is tracking down these corrupt
records and deleting them the best / only solution?
Is there a way to determine of there are issues with new data (after the
crash)?
Any help and advice is very much appreciated.
Thanks,
Nick Renders
On 5 Feb 2020, at 12:51, Alvaro Herrera wrote:
On 2020-Feb-05, Nick Renders wrote:
Is there anything specific I should check in our postgres
installation /
database to make sure it is running ok now? Anyway to see what the
consequences were of purging that one pg_clog file?
Losing pg_clog files is pretty bad, and should not happen; then again,
this might have been something else (ie. the file was maybe not lost).
That said, wrongly overwriting files is even worse.
By zeroing an existing pg_clog file, you marked a bunch of
transactions
as aborted. Your data is now probably inconsistent, if not downright
corrupt. I would be looking for my most recent backup ...
If you're very lucky, your database might be pg_dumpable. I would try
that, followed by restoring it in a separate clean instance and seeing
what happens.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
