Hot standby replication stalled

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Good morning,
I am running postgres 9.4.8.  We have two instances of production postgresql logshipping records to disaster recovery,then to replication development databases.  Things look good in the postgres logs and in the currency of transaction logs being log-shipped.  But for some inexplicable reason the replication has been stalled in one instance. The time of stall according to a simple query of a table that gets heavily updated looks to be around 4:35 am Saturday Oct 6. :
    
select max(last_change_dtm) from queuenodes;
            max             
----------------------------
 2018-10-06 04:35:58.799229
(1 row)

I have attached a snippet of the postgres logs documenting database activity around that time.
I have also attached the logshipping script that I use.  You will see that that script services the logshipping to the dr site (i/p 10.248.156.47) then to our developer replication site (i/p 10.93.156.52).  There have been no delays in log shipping:  we are current (Monday, Oct 8 at 9:45 am) so I am at a loss as to what is going on.  This has worked for about 8 months.

Another piece of information - we actively use pg_xlog_replay_pause() and pg_xlog_replay_resume() functions on our replication server but NOT on our dr server.  
I did run pg_is_xlog_replay_paused() on both servers to verify that the replay has not been paused.

Any help appreciated.
 -- 
Mark Steben
 Database Administrator
@utoRevenue | Autobase 
  CRM division of Dominion Dealer Solutions 
95D Ashley Ave.
West Springfield, MA 01089
t: 413.327-3045
f: 413.383-9567

www.fb.com/DominionDealerSolutions
www.twitter.com/DominionDealer
 www.drivedominion.com






ser@ardbc01 ~]$ less stagingstandby/copy.forhotstandby.legacy.sh 
#!/bin/sh
############################################################################
# $Author: cbrothers $
# $Date: 2015/12/15 11:16:34 $
# $Revision: 1.6 $
#
############################################################################
# Copyright (C) 2007 Dominion Enterprises All Rights Reserved
############################################################################
##  This script is invoked by the ARCHIVE_COMMAND parameter in the postgres
##  config file. It is run on server PROD_LEGACY_PSQL.SERVER.INT.
##         It will do the following:
##   
##   1. Copy the informational .backup and .history files to staging library
##   2. Gzip the actual xlogs to the staging library
##   3. Ping DR-LEGACY-PSQL.SERVER.INT to test for network connectivity
##   4. scp the following to DR-legacy-PSQL.SERVER.INT
##       -   informational log files 
##       -   the straight unzipped pg_log files for use in hot standby
##
##   The staging library on PROD-LEGACY-PSQL.SERVER.INT is:
##       /mnt/dbdumps/postgres-legacy/archlog
##
##  Copied to CVS FROM folder /MavMail/mm/InternalTools
##                                 /database_replication/ar-db-2desq
##
##  MES- 3/18/2015: if first testping successful, failover after 10 minutes
##     of unsuccessful scp
###########################################################################

REQ_FILE=$1
DEST=$2
GZDEST="${DEST}.gz"
LF="${REQ_FILE}.lock"
SUFFIX=${REQ_FILE##*.}
DATETIME=`date +"%Y%m%d-%H%M%S"`
PING='/bin/ping -c 1 -W 1'
##/usr/bin/lockfile "${LF}"  # lockfile also on receiving  server


#####################################################################
## logic to gzip xlogs to staging
####################################################################
if [ "${SUFFIX}" != 'history' ] && [ "${SUFFIX}" != 'backup' ]; then
  /bin/cat "${REQ_FILE}" | gzip > /mnt/dbdumps/postgres-legacy/archlog/${GZDEST}
  if [ "$?" -gt "0" ] ; then
    echo 'unsuccessful gzip of archlog file' >> /tmp/archlog.legacy.log
    echo 'the unsuccessful gzip exit code is '  "$?" >> /tmp/archlog.legacy.log
  fi
else
#####################################################################
## logic to copy informational files to staging
####################################################################
   cp "${REQ_FILE}"  "/mnt/dbdumps/postgres-legacy/archlog/${DEST}"
   if [ "$?" -gt "0" ] ; then
      echo " unsuccessful unzipped copy of backup info file" >> /tmp/archlog.legacy.log
      echo 'the unsuccessful backup file copy is ' "$?"  >> /tmp/archlog.legacy.log
   fi
fi
badping=0
${PING} 10.248.156.47
##########################################################################
## upon successful ping of dr-legacy-psql.server.int, scp to /shared/ardbc-dc3/archlog-legacy
##    folder on that server
##
##  Postgres should now will handle outages up to 30 minutes
##   if greater than 30 minutes dba should be called
#####################################################################
## 01/15/2015 - logic to scp current log directly to dr server - for hotstandby
##########################################################################
if [ "$?" = "0" ] ; then
 echo "the datetime before the scp to /shared/ardbc-dc3/archlog-legacy is " >> /tmp/archlog.legacy.log
  /bin/date  >> /tmp/archlog.legacy.log

  scp "${REQ_FILE}"  "pguser@xxxxxxxxxxxxxxxxxxxxxxxxx:/shared/ardbc-dc3/archlog-legacy/${DEST}"
 
  if [ "$?" != "0" ] ;  then
     echo "first log scp to dr-legacy-psql.server.int not good - trying again after sleeping 5 minutes. time is ${DATETIME}  " >> /tmp/archlog.legacy.log
     sleep 300
       scp "${REQ_FILE}"  "pguser@xxxxxxxxxxxxxxxxxxxxxxxxx:/shared/ardbc-dc3/archlog-legacy/${DEST}"
       if [ "$?" != "0" ] ;  then
         echo "second log scp to dr-legacy-psql.server.int not good - trying again after sleeping another 5 minutes. time is ${DATETIME} " >> /tmp/archlog.legacy.log
         sleep 300
         scp "${REQ_FILE}"  "pguser@xxxxxxxxxxxxxxxxxxxxxxxxx:/shared/ardbc-dc3/archlog-legacy/${DEST}"
         if  [ "$?" != "0" ] ;  then
           echo " third attempt at log scp to dr-legacy-psql.server.int failed - calling dba.  time is ${DATETIME}. log is ${DEST}  " >> /tmp/archlog.legacy.log 
           /MavMail/mm/InternalTools/system_tools/send_mail.pl 'markcell@xxxxxxxxxxxxxxx,ddsdba_autoalert@xxxxxxxxxxxxxxxxx' 'dbalert@xxxxxxxxxxxxxxxxxxxxxxxxxxx' 'dr-legacy-psql.server.int log scp unsuccessful after 3 attempts.' 'call m
ark 413-433-3124'
         fi
     fi 
  fi
 echo " the datetime after the scp to legacy /shared/ardbc-dc3/archlog-legacy folder is " >> /tmp/archlog.legacy.log
  /bin/date >> /tmp/archlog.legacy.log
else
  number=0
  badping=1
  echo "unsuccessful ping of dr-legacy-psql.server.int - first time - sleeping for 1 minute.  time is ${DATETIME}   " >> /tmp/archlog.legacy.log
  while [ "$badping" = '1' ] ; do
     echo "copying current archive log ${GZDEST} to holdinglegacy directory.      time is ${DATETIME} " >> /tmp/archlog.legacy.log
     /bin/cp -u  "/mnt/dbdumps/postgres-legacy/archlog/${GZDEST}" "/home/pguser/holdinglegacy/${GZDEST}"
     sleep 60
     number=$((number + 60))
     ${PING} 10.248.156.47
     if  [ "$?" = "0" ]   ;
     then
        echo " successful ping of dr-legacy-psql.server.int after $number seconds: sleeping for 20 seconds then continuing with scp copy from holdinglegacy. time is ${DATETIME} " >> /tmp/archlog.legacy.log
        sleep 20
        badping=0
        echo "the datetime before the scp to (dr-legacy)/shared/ardbc-dc3/archlog-legacy from the staging library is " >> /tmp/archlog.legacy.log
         /bin/date >> /tmp/archlog.legacy.log
         cd /home/pguser/holdinglegacy/
         gunzip *
         scp *  pguser@xxxxxxxxxxxxxxxxxxxxxxxxx:/shared/ardbc-dc3/archlog-legacy/ >> /tmp/archlog.legacy.log
         if  [ "$?" = "0" ]   ;
         then
             echo "the datetime after the successful scp from the staging library is " >> /tmp/archlog.legacy.log
             /bin/date >> /tmp/archlog.legacy.log
             /bin/rm /home/pguser/holdinglegacy/* -f
         else
             echo "unsuccessful scp to dr legacy after successful ping - sleeping for another minute " >> /tmp/archlog.legacy.log
         fi
     else
        echo " unsuccessful ping of prod-legacy-psql.server.int after $number seconds - sleeping for another minute. time is ${DATETIME} " >> /tmp/archlog.legacy.log
     fi
     if [ "$number" -ge "3600" ] ; then
        echo "  ping has been unsuccessful for more than 60 minutes - calling dba. time is ${DATETIME} " >> /tmp/archlog.legacy.log
            /MavMail/mm/InternalTools/system_tools/send_mail.pl 'markcell@xxxxxxxxxxxxxxx,ddsdba_autoalert@xxxxxxxxxxxxxxxxx' 'dbalert@xxxxxxxxxxxxxxxxxxxxxxxxxxx' 'prod-legacy-psql.server.int ping unsuccessful after 30 minutes'  'call mark 413-433-3124'
    fi
  done
fi
# 

sleep 60

badping=0
badscp=0
echo "this is the echo before the ping to arprodpsqlreplication01 " >> /tmp/archlog.legacy.toreplication.log

${PING} 10.93.156.52
##########################################################################
## upon successful ping of arprodpsqlreplication01, scp to 
##    /database/logship/PSQL_94/archlog-legacy
##
##  Postgres should now will handle outages up to 30 minutes
##   if greater than 30 minutes dba should be called
#####################################################################
## 01/15/2015 - logic to scp current log directly to dr server - for hotstandby
##########################################################################
if [ "$?" = "0" ] ; then
 echo "the datetime before the scp to /database/logship/PSQL_94/archlog-legacy is " >> /tmp/archlog.legacy.toreplication.log
  /bin/date  >> /tmp/archlog.legacy.toreplication.log

echo "this is the echo after the check of the ping to arprodpsqlreplication01 " >> /tmp/archlog.legacy.toreplication.log

  scp "${REQ_FILE}"  "pguser@arprodpsqlreplication01:/database/logship/PSQL_94/archlog-legacy/${DEST}"

if [ "$?" != "0" ] ;  then
     echo "the return code for first unsuccessful scp is  "$?" time is  ${DATETIME}  " >> /tmp/archlog.legacy.toreplication.log
     echo "first log scp to arprodpsqlreplication01 not good - trying again after sleeping 20 minutes. time is ${DATETIME}  " >> /tmp/archlog.legacy.toreplication.log
     sleep 1200
     badscp=1
  else
      echo " the log just copied is ${DEST}  " >> /tmp/archlog.legacy.toreplication.log
      echo " the datetime after the scp to arprodpsqlreplication01 legacy /database/logship/PSQL_94/archlog-legacy folder is   " >> /tmp/archlog.legacy.toreplication.log
      /bin/date >> /tmp/archlog.legacy.toreplication.log
     badscp=0
     exit 0
  fi

    scp "${REQ_FILE}"  "pguser@arprodpsqlreplication01:/database/logship/PSQL_94/archlog-legacy/${DEST}"
    if [ "$?" != "0" ] ;  then
       echo "the return code for second unsuccessful scp is  "$?" time is  ${DATETIME}  " >> /tmp/archlog.legacy.toreplication.log
       echo "second log scp to arprodpsqlreplication01 not good - trying again after sleeping another 20 minutes. time is ${DATETIME} " >> /tmp/archlog.legacy.toreplication.log
       sleep 1200
       badscp=1
    else
         echo " the log just copied is ${DEST}  " >> /tmp/archlog.legacy.toreplication.log
         echo " the datetime after the scp to arprodpsqlreplication01 legacy /database/logship/PSQL_94/archlog-legacy folder is " >> /tmp/archlog.legacy.toreplication.log
        /bin/date >> /tmp/archlog.legacy.toreplication.log
       badscp=0
       exit 0
    fi

    scp "${REQ_FILE}"  "pguser@arprodpsqlreplication01:/database/logship/PSQL_94/archlog-legacy/${DEST}"
     if  [ "$?" != "0" ] ;  then
         touch /home/pguser/legacy.logship.failure
        echo " third attempt at log scp to arprodpsqlreplication01 failed - calling dba.  time is ${DATETIME}. log is ${DEST}  " >> /tmp/archlog.legacy.toreplication.log
        /MavMail/mm/InternalTools/system_tools/send_mail.pl 'markcell@xxxxxxxxxxxxxxx,ddsdba_autoalert@xxxxxxxxxxxxxxxxx' 'dbalert@xxxxxxxxxxxxxxxxxxxxxxxxxxx' 'arprodpsqlreplication01 log scp unsuccessful after 3 attempts.' 'call mark 413-433-3124'
     else
       echo " the datetime after the scp to arprodpsqlreplication01 legacy /database/logship/PSQL_94/archlog-legacy folder is " >> /tmp/archlog.legacy.toreplication.log
       /bin/date >> /tmp/archlog.legacy.toreplication.log
       badscp=0
     fi
 
echo " the log just copied is ${DEST}  " >> /tmp/archlog.legacy.toreplication.log
echo " the datetime after the scp to arprodpsqlreplication01 legacy /database/logship/PSQL_94/archlog-legacy folder is " >> /tmp/archlog.legacy.toreplication.log
       /bin/date >> /tmp/archlog.legacy.toreplication.log

else
  number=0
  badping=1
  echo "unsuccessful ping of arprodpsqlreplication01 - first time - sleeping for 1 minute.  time is ${DATETIME}   " >> /tmp/archlog.legacy.toreplication.log
  while [ "$badping" = '1' ] ; do
     echo "copying current archive log ${GZDEST} to holdinglegacy directory.      time is ${DATETIME} " >> /tmp/archlog.legacy.toreplication.log
     /bin/cp -u  "/mnt/dbdumps/postgres-legacy/archlog/${GZDEST}" "/home/pguser/holdingreplegacy/${GZDEST}"
     sleep 60
     number=$((number + 60))
     ${PING} 10.93.156.52
     if  [ "$?" = "0" ]   ;
     then
        echo " successful ping of arprodpsqlreplication01 after $number seconds: sleeping for 20 seconds then continuing with scp copy from holdingreplegacy. time is ${DATETIME} " >> /tmp/archlog.legacy.toreplication.log
        sleep 20
        badping=0
        echo "the datetime before the scp to (arprodpsqlreplication01)/database/logship/PSQL_94/archlog-legacy from the staging library is " >> /tmp/archlog.legacy.toreplication.log
         /bin/date >> /tmp/archlog.legacy.toreplication.log
         cd /home/pguser/holdingreplegacy/
         gunzip *
         scp *  pguser@arprodpsqlreplication01:/database/logship/PSQL_94/archlog-legacy/ >> /tmp/archlog.legacy.toreplication.log
         if  [ "$?" = "0" ]   ;
         then
             echo "the datetime after the successful scp from the staging library is " >> /tmp/archlog.legacy.toreplication.log
             /bin/date >> /tmp/archlog.legacy.toreplication.log
             /bin/rm /home/pguser/holdingreplegacy/* -f
         else
             echo "unsuccessful scp to arprodpsqlrepl01 legacy after successful ping - sleeping for another minute " >> /tmp/archlog.legacy.toreplication.log
         fi
     else
        echo " unsuccessful ping of arprodpsqlreplication01 after $number seconds - sleeping for another minute. time is ${DATETIME} " >> /tmp/archlog.legacy.toreplication.log
     fi
     if [ "$number" -ge "3600" ] ; then
        echo "  ping has been unsuccessful for more than 60 minutes - calling dba. time is ${DATETIME} " >> /tmp/archlog.legacy.toreplication.log
            /MavMail/mm/InternalTools/system_tools/send_mail.pl 'markcell@xxxxxxxxxxxxxxx,ddsdba_autoalert@xxxxxxxxxxxxxxxxx' 'dbalert@xxxxxxxxxxxxxxxxxxxxxxxxxxx' 'arprodpsqlreplication01  ping unsuccessful after 30 minutes'  'call mark 413-433-3124'
    fi
  done
fi
# 

Oct  6 04:28:55 ardbc01dr postgres[60923]: [51842-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:00 ardbc01dr postgres[60923]: [51843-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:05 ardbc01dr postgres[60923]: [51844-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:10 ardbc01dr postgres[60923]: [51845-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:15 ardbc01dr postgres[60923]: [51846-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:20 ardbc01dr postgres[60923]: [51847-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:25 ardbc01dr postgres[60923]: [51848-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:30 ardbc01dr postgres[60923]: [51849-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:35 ardbc01dr postgres[60923]: [51850-1] @: LOG:  restored log file "0000000100007589000000C1" from archive
Oct  6 04:29:36 ardbc01dr postgres[60923]: [51851-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:29:36 ardbc01dr postgres[60923]: [51852-1] @: LOG:  unexpected pageaddr 7589/73000000 in log segment 0000000100007589000000C3, offset 0
Oct  6 04:29:40 ardbc01dr postgres[60923]: [51853-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:29:45 ardbc01dr postgres[60923]: [51854-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:29:50 ardbc01dr postgres[60923]: [51855-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:29:55 ardbc01dr postgres[60923]: [51856-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:00 ardbc01dr postgres[60923]: [51857-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:05 ardbc01dr postgres[60923]: [51858-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:10 ardbc01dr postgres[60923]: [51859-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:15 ardbc01dr postgres[60923]: [51860-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:20 ardbc01dr postgres[60923]: [51861-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:25 ardbc01dr postgres[60923]: [51862-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:31 ardbc01dr postgres[60923]: [51863-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:35 ardbc01dr postgres[60923]: [51864-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:40 ardbc01dr postgres[60923]: [51865-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:45 ardbc01dr postgres[60923]: [51866-1] @: LOG:  restored log file "0000000100007589000000C2" from archive
Oct  6 04:30:45 ardbc01dr postgres[60923]: [51867-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:30:45 ardbc01dr postgres[60923]: [51868-1] @: LOG:  unexpected pageaddr 7589/77000000 in log segment 0000000100007589000000C4, offset 0
Oct  6 04:30:50 ardbc01dr postgres[60923]: [51869-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:30:55 ardbc01dr postgres[60923]: [51870-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:00 ardbc01dr postgres[60923]: [51871-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:05 ardbc01dr postgres[60923]: [51872-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:10 ardbc01dr postgres[60923]: [51873-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:15 ardbc01dr postgres[60923]: [51874-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:20 ardbc01dr postgres[60923]: [51875-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:26 ardbc01dr postgres[60923]: [51876-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:30 ardbc01dr postgres[60923]: [51877-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:35 ardbc01dr postgres[60923]: [51878-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:40 ardbc01dr postgres[60923]: [51879-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:45 ardbc01dr postgres[60923]: [51880-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:50 ardbc01dr postgres[60923]: [51881-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:55 ardbc01dr postgres[60923]: [51882-1] @: LOG:  restored log file "0000000100007589000000C3" from archive
Oct  6 04:31:55 ardbc01dr postgres[60923]: [51883-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:31:55 ardbc01dr postgres[60923]: [51884-1] @: LOG:  unexpected pageaddr 7589/81000000 in log segment 0000000100007589000000C5, offset 0
Oct  6 04:32:01 ardbc01dr postgres[60923]: [51885-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:05 ardbc01dr postgres[60923]: [51886-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:10 ardbc01dr postgres[60923]: [51887-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:15 ardbc01dr postgres[60923]: [51888-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:20 ardbc01dr postgres[60923]: [51889-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:25 ardbc01dr postgres[60923]: [51890-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:30 ardbc01dr postgres[60923]: [51891-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:35 ardbc01dr postgres[60923]: [51892-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:40 ardbc01dr postgres[60923]: [51893-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:45 ardbc01dr postgres[60923]: [51894-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:50 ardbc01dr postgres[60923]: [51895-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:32:55 ardbc01dr postgres[60923]: [51896-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:33:00 ardbc01dr postgres[60923]: [51897-1] @: LOG:  restored log file "0000000100007589000000C4" from archive
Oct  6 04:33:00 ardbc01dr postgres[60923]: [51898-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:01 ardbc01dr postgres[60923]: [51899-1] @: LOG:  unexpected pageaddr 7589/7D000000 in log segment 0000000100007589000000C6, offset 0
Oct  6 04:33:05 ardbc01dr postgres[60923]: [51900-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:10 ardbc01dr postgres[60923]: [51901-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:15 ardbc01dr postgres[60923]: [51902-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:20 ardbc01dr postgres[60923]: [51903-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:25 ardbc01dr postgres[60923]: [51904-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:30 ardbc01dr postgres[60923]: [51905-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:35 ardbc01dr postgres[60923]: [51906-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:40 ardbc01dr postgres[60923]: [51907-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:45 ardbc01dr postgres[60923]: [51908-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:50 ardbc01dr postgres[60923]: [51909-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:33:55 ardbc01dr postgres[60923]: [51910-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:34:00 ardbc01dr postgres[60923]: [51911-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:34:05 ardbc01dr postgres[60923]: [51912-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:34:10 ardbc01dr postgres[60923]: [51913-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:34:15 ardbc01dr postgres[60923]: [51914-1] @: LOG:  restored log file "0000000100007589000000C5" from archive
Oct  6 04:34:16 ardbc01dr postgres[60923]: [51915-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:16 ardbc01dr postgres[60923]: [51916-1] @: LOG:  unexpected pageaddr 7589/7C000000 in log segment 0000000100007589000000C7, offset 0
Oct  6 04:34:20 ardbc01dr postgres[60923]: [51917-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:25 ardbc01dr postgres[60923]: [51918-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:30 ardbc01dr postgres[60923]: [51919-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:35 ardbc01dr postgres[60923]: [51920-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:40 ardbc01dr postgres[60923]: [51921-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:45 ardbc01dr postgres[60923]: [51922-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:50 ardbc01dr postgres[60923]: [51923-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:34:56 ardbc01dr postgres[60923]: [51924-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:00 ardbc01dr postgres[60923]: [51925-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:05 ardbc01dr postgres[60923]: [51926-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:10 ardbc01dr postgres[60923]: [51927-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:15 ardbc01dr postgres[60923]: [51928-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:20 ardbc01dr postgres[60923]: [51929-1] @: LOG:  restored log file "0000000100007589000000C6" from archive
Oct  6 04:35:20 ardbc01dr postgres[60923]: [51930-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:20 ardbc01dr postgres[60923]: [51931-1] @: LOG:  unexpected pageaddr 7589/74000000 in log segment 0000000100007589000000C8, offset 0
Oct  6 04:35:25 ardbc01dr postgres[60923]: [51932-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:30 ardbc01dr postgres[60923]: [51933-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:35 ardbc01dr postgres[60923]: [51934-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:40 ardbc01dr postgres[60923]: [51935-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:45 ardbc01dr postgres[60923]: [51936-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:50 ardbc01dr postgres[60923]: [51937-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:35:55 ardbc01dr postgres[60923]: [51938-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:00 ardbc01dr postgres[60923]: [51939-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:05 ardbc01dr postgres[60923]: [51940-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:10 ardbc01dr postgres[60923]: [51941-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:15 ardbc01dr postgres[60923]: [51942-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:20 ardbc01dr postgres[60923]: [51943-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:25 ardbc01dr postgres[60923]: [51944-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:30 ardbc01dr postgres[60923]: [51945-1] @: LOG:  restored log file "0000000100007589000000C7" from archive
Oct  6 04:36:31 ardbc01dr postgres[60923]: [51946-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:31 ardbc01dr postgres[60923]: [51947-1] @: LOG:  unexpected pageaddr 7589/72000000 in log segment 0000000100007589000000C9, offset 0
Oct  6 04:36:35 ardbc01dr postgres[60923]: [51948-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:40 ardbc01dr postgres[60923]: [51949-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:45 ardbc01dr postgres[60923]: [51950-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:50 ardbc01dr postgres[60923]: [51951-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:55 ardbc01dr postgres[60923]: [51952-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:00 ardbc01dr postgres[60923]: [51953-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:05 ardbc01dr postgres[60923]: [51954-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:10 ardbc01dr postgres[60923]: [51955-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:15 ardbc01dr postgres[60923]: [51956-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:20 ardbc01dr postgres[60923]: [51957-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:25 ardbc01dr postgres[60923]: [51958-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:30 ardbc01dr postgres[60923]: [51959-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:35 ardbc01dr postgres[60923]: [51960-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:40 ardbc01dr postgres[60923]: [51961-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:41 ardbc01dr postgres[60923]: [51962-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:37:41 ardbc01dr postgres[60923]: [51963-1] @: LOG:  unexpected pageaddr 7589/80000000 in log segment 0000000100007589000000CA, offset 0
Oct  6 04:37:45 ardbc01dr postgres[60923]: [51964-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:36:40 ardbc01dr postgres[60923]: [51949-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:45 ardbc01dr postgres[60923]: [51950-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:50 ardbc01dr postgres[60923]: [51951-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:36:55 ardbc01dr postgres[60923]: [51952-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:00 ardbc01dr postgres[60923]: [51953-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:05 ardbc01dr postgres[60923]: [51954-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:10 ardbc01dr postgres[60923]: [51955-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:15 ardbc01dr postgres[60923]: [51956-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:20 ardbc01dr postgres[60923]: [51957-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:25 ardbc01dr postgres[60923]: [51958-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:30 ardbc01dr postgres[60923]: [51959-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:35 ardbc01dr postgres[60923]: [51960-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:40 ardbc01dr postgres[60923]: [51961-1] @: LOG:  restored log file "0000000100007589000000C8" from archive
Oct  6 04:37:41 ardbc01dr postgres[60923]: [51962-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:37:41 ardbc01dr postgres[60923]: [51963-1] @: LOG:  unexpected pageaddr 7589/80000000 in log segment 0000000100007589000000CA, offset 0
Oct  6 04:37:45 ardbc01dr postgres[60923]: [51964-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:37:50 ardbc01dr postgres[60923]: [51965-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:37:55 ardbc01dr postgres[60923]: [51966-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:00 ardbc01dr postgres[60923]: [51967-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:05 ardbc01dr postgres[60923]: [51968-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:10 ardbc01dr postgres[60923]: [51969-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:15 ardbc01dr postgres[60923]: [51970-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:20 ardbc01dr postgres[60923]: [51971-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:25 ardbc01dr postgres[60923]: [51972-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:30 ardbc01dr postgres[60923]: [51973-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:35 ardbc01dr postgres[60923]: [51974-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:40 ardbc01dr postgres[60923]: [51975-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:45 ardbc01dr postgres[60923]: [51976-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:50 ardbc01dr postgres[60923]: [51977-1] @: LOG:  restored log file "0000000100007589000000C9" from archive
Oct  6 04:38:51 ardbc01dr postgres[60923]: [51978-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:38:51 ardbc01dr postgres[60923]: [51979-1] @: LOG:  unexpected pageaddr 7589/71000000 in log segment 0000000100007589000000CB, offset 0
Oct  6 04:38:55 ardbc01dr postgres[60923]: [51980-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:00 ardbc01dr postgres[60923]: [51981-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:05 ardbc01dr postgres[60923]: [51982-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:10 ardbc01dr postgres[60923]: [51983-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:15 ardbc01dr postgres[60923]: [51984-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:20 ardbc01dr postgres[60923]: [51985-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:25 ardbc01dr postgres[60923]: [51986-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:30 ardbc01dr postgres[60923]: [51987-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:35 ardbc01dr postgres[60923]: [51988-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:40 ardbc01dr postgres[60923]: [51989-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:45 ardbc01dr postgres[60923]: [51990-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:50 ardbc01dr postgres[60923]: [51991-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
Oct  6 04:39:55 ardbc01dr postgres[60923]: [51992-1] @: LOG:  restored log file "0000000100007589000000CA" from archive
[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux