Re: 9.0.4 Data corruption issue

[Ken Caruso <ken@xxxxxxxxx>]

Forgot to mention we have streaming replication partner as well that around the same time stop recovering because a WAL segment could not be found on the primary.

On Sat, Jul 16, 2011 at 4:05 AM, Ken Caruso <ken@xxxxxxxxx> wrote:

Hello,

Had a DB that had bloated due to lots of updates and insuffcient auto vacuum settings. Attempted the following in the following order:

VACUUM VERBOSE full tablename;

CLUSTER tablename; 

Did the full first to ensure that the cluster had enough space to run. 

The db had already been previously clustered to an index. At some point during the VACUUM I started seeing the following errors in the logs:

 CONTEXT:  writing block 12125253 of relation base/2651908/652397108

 WARNING:  could not write block 12125253 of base/2651908/652397108

 DETAIL:  Multiple failures --- write error might be permanent.

Also seeing the same error for block 12125253,  2978. 

And the cluster failed as well. I have attempted a check point using pg_start_backup() and that failed as well. 

The system is still running and I have yet to shut it down. I am hoping to find out what possibilities I have if any to recover from this. Also how do I avoid this in the future?

-Ken