Hello Everyone,
I am looking for suggestion how to recover bdr replication.
The short story we have 2 virtual nodes with share storage. Share storage lost power and after I brought all online bdr doesn't work properly.
Here are some log
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] LOG: starting logical decoding for slot "bdr_16386_6242730220644137235_1_17937__"
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] DETAIL: streaming transactions committing after 0/39A248B0, reading WAL from 0/39A248B0
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] LOG: starting logical decoding for slot "bdr_20650_6242730220644137235_1_22220__"
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] DETAIL: streaming transactions committing after 0/39A248B0, reading WAL from 0/39A248B0
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] ERROR: requested WAL segment 000000010000000000000039 has already been removed
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] ERROR: requested WAL segment 000000010000000000000039 has already been removed
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] LOG: could not receive data from client: Connection reset by peer
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] LOG: could not receive data from client: Connection reset by peer
2016-03-30 20:47:28 EDT @: [977] LOG: starting background worker process "bdr (6242981939774453042,1,20650,)->bdr (6242730220644137235,1,"
2016-03-30 20:47:28 EDT @: [977] LOG: starting background worker process "bdr (6242981939774453042,1,16386,)->bdr (6242730220644137235,1,"
2016-03-30 20:47:28 EDT @: [15899] NOTICE: version "1.0" of extension "btree_gist" is already installed
2016-03-30 20:47:28 EDT @: [15899] NOTICE: version "0.9.3.0" of extension "bdr" is already installed
2016-03-30 20:47:28 EDT @: [15898] NOTICE: version "1.0" of extension "btree_gist" is already installed
2016-03-30 20:47:28 EDT @: [15898] NOTICE: version "0.9.3.0" of extension "bdr" is already installed
2016-03-30 20:47:28 EDT @: [15899] INFO: starting up replication from 1 at 0/0
2016-03-30 20:47:28 EDT @: [15898] INFO: starting up replication from 8 at 0/0
2016-03-30 20:47:28 EDT @: [15899] ERROR: data stream ended
2016-03-30 20:47:28 EDT @: [15898] ERROR: data stream ended
2016-03-30 20:47:28 EDT @: [977] LOG: worker process: bdr (6242981939774453042,1,20650,)->bdr (6242730220644137235,1, (PID 15898) exited with exit code 1
2016-03-30 20:47:28 EDT @: [977] LOG: worker process: bdr (6242981939774453042,1,16386,)->bdr (6242730220644137235,1, (PID 15899) exited with exit code 1
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] DETAIL: streaming transactions committing after 0/39A248B0, reading WAL from 0/39A248B0
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] LOG: starting logical decoding for slot "bdr_20650_6242730220644137235_1_22220__"
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] DETAIL: streaming transactions committing after 0/39A248B0, reading WAL from 0/39A248B0
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] ERROR: requested WAL segment 000000010000000000000039 has already been removed
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] ERROR: requested WAL segment 000000010000000000000039 has already been removed
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43712):fusionpbx_prod01 [15896] LOG: could not receive data from client: Connection reset by peer
2016-03-30 20:47:26 EDT postgres@10.12.160.2(43714):freeswitch_prod01 [15897] LOG: could not receive data from client: Connection reset by peer
2016-03-30 20:47:28 EDT @: [977] LOG: starting background worker process "bdr (6242981939774453042,1,20650,)->bdr (6242730220644137235,1,"
2016-03-30 20:47:28 EDT @: [977] LOG: starting background worker process "bdr (6242981939774453042,1,16386,)->bdr (6242730220644137235,1,"
2016-03-30 20:47:28 EDT @: [15899] NOTICE: version "1.0" of extension "btree_gist" is already installed
2016-03-30 20:47:28 EDT @: [15899] NOTICE: version "0.9.3.0" of extension "bdr" is already installed
2016-03-30 20:47:28 EDT @: [15898] NOTICE: version "1.0" of extension "btree_gist" is already installed
2016-03-30 20:47:28 EDT @: [15898] NOTICE: version "0.9.3.0" of extension "bdr" is already installed
2016-03-30 20:47:28 EDT @: [15899] INFO: starting up replication from 1 at 0/0
2016-03-30 20:47:28 EDT @: [15898] INFO: starting up replication from 8 at 0/0
2016-03-30 20:47:28 EDT @: [15899] ERROR: data stream ended
2016-03-30 20:47:28 EDT @: [15898] ERROR: data stream ended
2016-03-30 20:47:28 EDT @: [977] LOG: worker process: bdr (6242981939774453042,1,20650,)->bdr (6242730220644137235,1, (PID 15898) exited with exit code 1
2016-03-30 20:47:28 EDT @: [977] LOG: worker process: bdr (6242981939774453042,1,16386,)->bdr (6242730220644137235,1, (PID 15899) exited with exit code 1
