Recovery Standby Node

[Chander Jamwal <silvi.jamwal@xxxxxxxxx>]

Hi,

My standby node is not getting sync with master node. I am performing following steps:

1. $ rm -rf /opt/pg/data/9.4/*
2. $ pg_basebackup -h <master-node-ip> -p 1523 -U pg -D /opt/pg/data/9.4 -X stream -P
3. rm -rf /opt/pg/tmp/PGSQL.lock
4.  pcs resource cleanup msPostgresql   //msPostgresql is resource name

Following are the log:

pgsql(pgsql)[5679]:     2018/02/12_02:38:25 INFO: PostgreSQL is down
pgsql(pgsql)[5679]:     2018/02/12_02:38:26 INFO: PostgreSQL is down
Feb 12 02:38:27 [5562] zjedmsdp20v       lrmd:  warning: child_timeout_callback:        pgsql_start_0 process (PID 5679) timed out
Feb 12 02:38:27 [5562] zjedmsdp20v       lrmd:  warning: operation_finished:    pgsql_start_0:5679 - timed out after 60000ms
Feb 12 02:38:27 [5562] zjedmsdp20v       lrmd:     info: log_finished:  finished - rsc:pgsql action:start call_id:11 pid:5679 exit-code:1 exec-time:60001ms queue-time:0ms
Feb 12 02:38:27 [5565] zjedmsdp20v       crmd:     info: services_os_action_execute:    Managed pgsql_meta-data_0 process 6302 exited with rc=0
Feb 12 02:38:27 [5565] zjedmsdp20v       crmd:    error: process_lrm_event:     Operation pgsql_start_0: Timed Out (node=zjedmsdp20v, call=11, timeout=60000ms)
Feb 12 02:38:27 [5560] zjedmsdp20v        cib:     info: cib_process_request:   Forwarding cib_modify operation for section status to master (origin=local/crmd/12)
Feb 12 02:38:27 [5560] zjedmsdp20v        cib:     info: cib_perform_op:        Diff: --- 0.122.31 2
Feb 12 02:38:27 [5560] zjedmsdp20v        cib:     info: cib_perform_op:        Diff: +++ 0.122.32 (null)
Feb 12 02:38:27 [5560] zjedmsdp20v        cib:     info: cib_perform_op:        +  /cib:  @num_updates=32
Feb 12 02:38:27 [5560] zjedmsdp20v        cib:     info: cib_perform_op:        +  /cib/status/node_state[@id='zjedmsdp20v']/lrm[@id='zjedmsdp20v']/lrm_resources/lrm_resource[@id='pgsql']/lrm
_rsc_op[@id='pgsql_last_0']:  @operation_key=pgsql_start_0, @operation=start, @transition-key=15:7444:0:8c793138-d4e4-4780-88f2-2da6808c25ca, @transition-magic=2:1;15:7444:0:8c793138-d4e4-478
0-88f2-2da6808c25ca, @call-id=11, @rc-code=1, @op-status=2, @exec-time=60001

And output of crm_mod -Afr -1 is:

Last updated: Mon Feb 12 02:39:00 2018
Last change: Mon Feb 12 02:31:45 2018
Stack: cman
Current DC: zjedmsdp19v - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
3 Resources configured


Online: [ zjedmsdp19v zjedmsdp20v ]

Full list of resources:

 vip-master    (ocf::heartbeat:IPaddr2):    Started zjedmsdp19v
 Master/Slave Set: msPostgresql [pgsql]
     Masters: [ zjedmsdp19v ]
     Stopped: [ zjedmsdp20v ]

Node Attributes:
* Node zjedmsdp19v:
    + master-pgsql                        : 1000     
    + pgsql-data-status                   : LATEST   
    + pgsql-master-baseline               : 0000002054000090
    + pgsql-status                        : PRI      
* Node zjedmsdp20v:
    + master-pgsql                        : -INFINITY
    + pgsql-data-status                   : DISCONNECT
    + pgsql-status                        : STOP     

Migration summary:
* Node zjedmsdp19v:
* Node zjedmsdp20v:
   pgsql: migration-threshold=1 fail-count=1000000 last-failure='Mon Feb 12 02:38:27 2018'

Failed actions:
    pgsql_start_0 on zjedmsdp20v 'unknown error' (1): call=11, status=Timed Out, last-rc-change='Mon Feb 12 02:37:27 2018', queued=0ms, exec=60001ms

I shall thankful any guidance or support on this

Regards,

Chander