|
Hi I have setup a master/standby on PostgreSQL95 on two test servers and trialing out repmgr. (https://github.com/2ndQuadrant/repmgr/)
I am testing a switchover using the following: -bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf -C /etc/repmgr/9.5/repmgr.conf standby switchover -L DEBUG -v The switchover appears to hang at the last part of the switchover process…. NOTICE: restarting server using '/usr/pgsql-9.5/bin/pg_ctl -w -D /var/lib/pgsql/9.5/data -m fast restart' pg_ctl: PID file "/var/lib/pgsql/9.5/data/postmaster.pid" does not exist Is server running? starting server anyway It appears to have worked though as when I run the cluster show command on both servers it showing the switchover. -bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf cluster show Role | Name | Upstream | Connection String ----------+----------------|----------------|------------------------------------------- * master | itupl-postgen2 | | host=10.70.3.252 dbname=repmgr user=repmgr standby | itupl-postgen1 | itupl-postgen2 | host=10.70.3.251 dbname=repmgr user=repmgr It is also showing correctly in repl_nodes table of the two databases.
Why is it hanging?? Thank you for your help… Here is the complete output: -----------------------------------------------
-bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf -C /etc/repmgr/9.5/repmgr.conf standby switchover -L DEBUG -v NOTICE: using configuration file "/etc/repmgr/9.5/repmgr.conf" NOTICE: switching current node 2 to master server and demoting current master to standby... DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr fallback_application_name='repmgr'' DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() INFO: retrieving node list for cluster 'repmgr_cluster' DEBUG: get_master_connection(): SELECT id, conninfo, CASE WHEN type = 'master' THEN 1 ELSE 2 END AS type_priority FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster = 'repmgr_cluster' AND type != 'witness' ORDER BY
active DESC, type_priority, priority, id INFO: checking role of cluster node '1' DEBUG: connecting to: 'host=10.70.3.251 dbname=repmgr user=repmgr fallback_application_name='repmgr'' DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() DEBUG: get_master_connection(): current master node is 1 DEBUG: get_node_record(): SELECT id, type, upstream_node_id, name, conninfo, slot_name, priority, active FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster = 'repmgr_cluster' AND id = 1 DEBUG: remote node name is "itupl-postgen1" DEBUG: test_ssh_connection(): executing ssh -o Batchmode=yes 10.70.3.251 /bin/true 2>/dev/null DEBUG: get_pg_setting(): SELECT name, setting FROM pg_catalog.pg_settings WHERE name = 'data_directory' DEBUG: get_pg_setting(): returned value is "/var/lib/pgsql/9.5/data" DEBUG: master's data directory is: /var/lib/pgsql/9.5/data DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls '/var/lib/pgsql/9.5/data/PG_VERSION' >/dev/null 2>&1 && echo 1 || echo 0 DEBUG: remote_command(): output returned was: 1 DEBUG: PG_VERSION found in /var/lib/pgsql/9.5/data DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls '/usr/pgsql-9.5/bin/pg_rewind' >/dev/null 2>&1 && echo 1 || echo 0 DEBUG: remote_command(): output returned was: 1 DEBUG: guc_set(): SELECT true FROM pg_catalog.pg_settings WHERE name = 'full_page_writes' AND setting = 'off' DEBUG: guc_set(): SELECT true FROM pg_catalog.pg_settings WHERE name = 'wal_log_hints' AND setting = 'on' INFO: looking for file "/etc/repmgr/9.5/repmgr.conf" on remote server "10.70.3.251" DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls '/etc/repmgr/9.5/repmgr.conf' >/dev/null 2>&1 && echo 1 || echo 0 DEBUG: remote_command(): output returned was: 1 INFO: remote configuration file "/etc/repmgr/9.5/repmgr.conf" found on remote server DEBUG: remote_archive_config_dir: /tmp/repmgr-itupl-postgen1-archive DEBUG: Executing: /usr/pgsql-9.5/bin/repmgr standby archive-config -f '/etc/repmgr/9.5/repmgr.conf' --config-archive-dir='/tmp/repmgr-itupl-postgen1-archive' DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 /usr/pgsql-9.5/bin/repmgr standby archive-config -f '/etc/repmgr/9.5/repmgr.conf' --config-archive-dir='/tmp/repmgr-itupl-postgen1-archive' WARNING: nonstandard use of escape in a string literal LINE 1: ...config_file, regexp_replace(config_file, '^.*\/',''... ^ HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. NOTICE: 3 files copied to /tmp/repmgr-itupl-postgen1-archive DEBUG: remote_command(): output returned was: DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 /usr/pgsql-9.5/bin/pg_ctl -D '/var/lib/pgsql/9.5/data' -m fast -W stop >/dev/null 2>&1 && echo 1 || echo 0 DEBUG: remote_command(): output returned was: 1 DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls '/var/lib/pgsql/9.5/data/postmaster.pid' >/dev/null 2>&1 && echo 1 || echo 0 DEBUG: remote_command(): output returned was: 0 NOTICE: current master has been stopped INFO: connecting to standby database DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr fallback_application_name='repmgr'' INFO: connected to standby, checking its state DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() INFO: retrieving node list for cluster 'repmgr_cluster' DEBUG: get_master_connection(): SELECT id, conninfo, CASE WHEN type = 'master' THEN 1 ELSE 2 END AS type_priority FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster = 'repmgr_cluster' AND type != 'witness' ORDER BY
active DESC, type_priority, priority, id INFO: checking role of cluster node '1' DEBUG: connecting to: 'host=10.70.3.251 dbname=repmgr user=repmgr fallback_application_name='repmgr'' ERROR: connection to database failed: could not connect to server: Connection refused Is the server running on host "10.70.3.251" and accepting TCP/IP connections on port 5432? INFO: checking role of cluster node '2' DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr fallback_application_name='repmgr'' DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() NOTICE: promoting standby DEBUG: get_pg_setting(): SELECT name, setting FROM pg_catalog.pg_settings WHERE name = 'data_directory' DEBUG: get_pg_setting(): returned value is "/var/lib/pgsql/9.5/data" NOTICE: promoting server using '/usr/pgsql-9.5/bin/pg_ctl -D /var/lib/pgsql/9.5/data promote' server promoting INFO: reconnecting to promoted server DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr fallback_application_name='repmgr'' DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery() DEBUG: setting node 2 as master and marking existing master as failed DEBUG: begin_transaction() DEBUG: commit_transaction() NOTICE: STANDBY PROMOTE successful DEBUG: create_event_record(): INSERT INTO "repmgr_repmgr_cluster".repl_events ( node_id, event, successful, details ) VALUES ($1, $2, $3, $4) RETURNING event_timestamp DEBUG: create_event_record(): Event timestamp is "2017-05-22 16:56:06.860066+09:30" NOTICE: Executing pg_rewind on old master server DEBUG: pg_rewind command is: '/usr/pgsql-9.5/bin/pg_rewind' -D '/var/lib/pgsql/9.5/data' --source-server=\'host=10.70.3.252 dbname=repmgr user=repmgr\' DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 '/usr/pgsql-9.5/bin/pg_rewind' -D '/var/lib/pgsql/9.5/data' --source-server=\'host=10.70.3.252 dbname=repmgr user=repmgr\' DEBUG: remote_command(): output returned was: servers diverged at WAL position 1/1D000098 on timeline 11 no rewind required DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 /usr/pgsql-9.5/bin/repmgr standby restore-config -D '/var/lib/pgsql/9.5/data' --config-archive-dir='/tmp/repmgr-itupl-postgen1-archive' ERROR: unable to determine cluster name - please provide a valid configuration file with -c/--config-file HINT: Use -F/--force to continue anyway DEBUG: remote_command(): output returned was: DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 test -e '/var/lib/pgsql/9.5/data/recovery.done' && rm -f '/var/lib/pgsql/9.5/data/recovery.done' DEBUG: remote_command(): output returned was: DEBUG: Executing: /usr/pgsql-9.5/bin/repmgr -D '/var/lib/pgsql/9.5/data' -f '/etc/repmgr/9.5/repmgr.conf' -h 10.70.3.252 -d repmgr -U repmgr standby follow DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 /usr/pgsql-9.5/bin/repmgr -D '/var/lib/pgsql/9.5/data' -f '/etc/repmgr/9.5/repmgr.conf' -h 10.70.3.252 -d repmgr -U repmgr standby follow NOTICE: restarting server using '/usr/pgsql-9.5/bin/pg_ctl -w -D /var/lib/pgsql/9.5/data -m fast restart' pg_ctl: PID file "/var/lib/pgsql/9.5/data/postmaster.pid" does not exist Is server running? starting server anyway Regards Dylan |
