Re: Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

[desmodemone <desmodemone@xxxxxxxxx>]

               

2014-02-10 12:29 GMT+01:00 Cassiano, Marco <mcassiano@xxxxxxxxxx>:

I resend the mail with gzipped attachment due to mailing list message size limits

--------------------------

 

Thank you Mat,

 

here are the additional infos :

 

1)       All of the three servers (main+2 replicas) are virtual on VMware Esxi 5.0

2)       Each server is on a different storage and on different vmware hosts

3)       The Log of the primary are with no errors

4)       Attached : pg_controldata output, postgres log, and /var/log/messages

5)       Fsck on the colume containing the database folders reports no error :

 

[root@pg64prod_rep /]# umount /dev/sdb1

[root@pg64prod_rep /]# fsck -n /dev/sdb1

fsck from util-linux-ng 2.17.2

e2fsck 1.41.12 (17-May-2010)

/dev/sdb1 has gone 193 days without being checked, check forced.

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

/dev/sdb1: 7886/13107200 files (8.3% non-contiguous), 38818825/52428119 blocks

 

Marco

 

Da: desmodemone [mailto:desmodemone@xxxxxxxxx]

Inviato: lunedì 10 febbraio 2014 10:30
A: Cassiano, Marco
Cc: pgsql-admin@xxxxxxxxxxxxxx

Oggetto: Re: Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

 

 

 

2014-02-10 9:30 GMT+01:00 Cassiano, Marco <mcassiano@xxxxxxxxxx>:

 

Hello,

           please, could you post some details about?  Your replica are on the same storage  as the primary ? Are virtual or physical ? hypervisor type?

Could you attach / post /var/log/messages and postgres log  ?

Could you attach / post   pg_controldata output of the replica ?

Did you verify the filesystem integrity of the replica ?

 

Are the log of the primary without errors ?

 

Thank you very much

Mat

 

Hi Marco,

Thanks for your log files about the problem and sorry for late answer,  in the source code of xlogutils.c I see that more log information are logged at DEBUG2 level of

log_min_messages <= DEBUG2 || client_min_messages <= DEBUG2

So I  think will be a good idea to modify your postgresql.conf of one standby at set the log_min_messages= DEBUG2 , so the code will invoke the function report_invalid_page and more information
will be reported about the invalid page of the relation on which investigate.

Bye

Mat