Re: Tracking down log segment corruption

[Gordon Shannon <gordo169@xxxxxxxxx>]

On Sun, May 2, 2010 at 12:10 PM, Tom Lane <tgl@xxxxxxxxxxxxx> wrote:

No, this would be a pg_database row with that OID.  But it looks like
you found the relevant index anyway.

Yup, realized that on second reading.
 

> These commands worked fine on the master, yet this seems suspiciously
> relevant.

Yeah, perhaps so.  What time did the failure on the standby occur (how
long after you did those moves)?  Is it reasonable to assume that this
was the first subsequent access to the index?

Bingo.  Yes it is reasonable.  It was 25 seconds between my altering the index in question and the server crash.

My local commands (in MDT, plus my machine is 15 sec ahead of the server):

09:10:52> alter index cts_20100501_natural_uk set tablespace ts30;
ALTER INDEX
Time: 787.790 ms
09:11:41> alter index cts_20100501_pkey set tablespace ts30;
ALTER INDEX
Time: 468.526 ms
09:11:51> alter index cts_20100501_topic_date_nk set tablespace ts30;
ALTER INDEX
Time: 385.322 ms
09:11:59> alter index cts_20100501_updated_nk set tablespace ts30;
ALTER INDEX
Time: 963.150 ms
09:12:10> alter table cts_20100501 set tablespace ts29;
ALTER TABLE

And from the wsb log (times in EDT):

4158   2010-05-02 11:12:09 EDT [26446]LOG:  restored log file "0000000100003C77000000C4" from archive
4158   2010-05-02 11:12:09 EDT [26447]WARNING:  specified item offset is too large
4158   2010-05-02 11:12:09 EDT [26448]CONTEXT:  xlog redo insert: rel 48777166/22362/48778276; tid 2/2
4158   2010-05-02 11:12:09 EDT [26449]PANIC:  btree_insert_redo: failed to add item