Greetings,
We recently experienced two similar outages on two different prod databases. The error messages from the logs were as follows:
LOG: server process (PID 138529) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
We are still investigating what may have triggered these errors, since there were no recent changes to these databases. Unfortunately, core dumps were not configured correctly, so we may have to wait for the next outage before we can do a good root cause analysis.
My question, meanwhile, is around remedial actions to take when this happens.
In one case, the logs recorded
LOG: all server processes terminated; reinitializing
LOG: incomplete data in "postmaster.pid": found only 1 newlines while trying to add line 7
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 365/FDFA738
LOG: invalid record length at 365/12420978: wanted 24, got 0
LOG: redo done at 365/12420950
LOG: last completed transaction was at log time 2018-06-05 10:59:27.049443-05
LOG: checkpoint starting: end-of-recovery immediate
LOG: checkpoint complete: wrote 5343 buffers (0.5%); 0 transaction log file(s) added, 1 removed, 0 recycled; write=0.131 s, sync=0.009 s, total=0.164 s; sync files=142, longest=0.005 s, average=0.000 s; distance=39064 kB, estimate=39064 kB
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
In that case, the database restarted immediately, with only 30 seconds of downtime.
In the other case, the logs recorded
LOG: all server processes terminated; reinitializing
LOG: dynamic shared memory control segment is corrupt
LOG: incomplete data in "postmaster.pid": found only 1 newlines while trying to add line 7
In that case, the database did not restart on its own. It was 5 am on Sunday, so the on-call SRE just manually started the database up, and it appears to have been running fine since.
LOG: all server processes terminated; reinitializing
LOG: incomplete data in "postmaster.pid": found only 1 newlines while trying to add line 7
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 365/FDFA738
LOG: invalid record length at 365/12420978: wanted 24, got 0
LOG: redo done at 365/12420950
LOG: last completed transaction was at log time 2018-06-05 10:59:27.049443-05
LOG: checkpoint starting: end-of-recovery immediate
LOG: checkpoint complete: wrote 5343 buffers (0.5%); 0 transaction log file(s) added, 1 removed, 0 recycled; write=0.131 s, sync=0.009 s, total=0.164 s; sync files=142, longest=0.005 s, average=0.000 s; distance=39064 kB, estimate=39064 kB
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
In that case, the database restarted immediately, with only 30 seconds of downtime.
In the other case, the logs recorded
LOG: all server processes terminated; reinitializing
LOG: dynamic shared memory control segment is corrupt
LOG: incomplete data in "postmaster.pid": found only 1 newlines while trying to add line 7
In that case, the database did not restart on its own. It was 5 am on Sunday, so the on-call SRE just manually started the database up, and it appears to have been running fine since.
My question is whether the corrupt shared memory control segment, and the failure of Postgres to automatically restart, mean the database should not be automatically started up, and if there's something we should be doing before restarting.
Do we potentially have corrupt data or indices as a result of our last outage? If so, what should we do to investigate?
Thanks,
Sherrylyn
