Re: Critical errors during logical decoding

[Colin Morelli <colin.morelli@xxxxxxxxx>]

Appears to be a problem with pglogical_output when de-toasting columns. Switched to wal2json and the problem went away. I believe I may have found the issue anyway. Apologies for the noise.

Best,

Colin

On Wed, Feb 7, 2018 at 2:20 PM, Colin Morelli <colin.morelli@xxxxxxxxx> wrote:

List,

For seemingly no reason at all, my logical replication slot has just started bailing out with errors every time my client tries to connect to it. Particularly - at this moment - I'm getting:

2018-02-07 19:14:31 UTC [3985-1] svc_app@app LOG:  00000: received replication command: START_REPLICATION SLOT event_stream LOGICAL 0/0 ("startup_params_format" '1', "no_txinfo" 'true', "expected_encoding" 'UTF8', "max_proto_version" '1', "proto_format" 'json', "min_proto_version" '1')

2018-02-07 19:14:31 UTC [3985-2] svc_app@app LOCATION:  exec_replication_command, walsender.c:1341

2018-02-07 19:14:31 UTC [3985-3] svc_app@app LOG:  00000: starting logical decoding for slot "event_stream"

2018-02-07 19:14:31 UTC [3985-4] svc_app@app DETAIL:  streaming transactions committing after 0/8DDFD48, reading WAL from 0/8DDDC00

2018-02-07 19:14:31 UTC [3985-5] svc_app@app LOCATION:  CreateDecodingContext, logical.c:408

2018-02-07 19:14:31 UTC [3985-6] svc_app@app LOG:  00000: logical decoding found consistent point at 0/8DDDC00

2018-02-07 19:14:31 UTC [3985-7] svc_app@app DETAIL:  There are no running transactions.

2018-02-07 19:14:31 UTC [3985-8] svc_app@app LOCATION:  SnapBuildFindSnapshot, snapbuild.c:1245

2018-02-07 19:14:31 UTC [3985-9] svc_app@app ERROR:  XX000: no known snapshots

Other errors have included missing toast entries, as well as missing base/ files on disk. I blew away the entire data directory, and restored the database from a pg_dump to a fresh directory after checking filesystem consistency. The logical replication stream operates for a few minutes, and then eventually bails out again. This is Postgres 9.6.6 on Ubuntu installed from packages.

Does anyone have any insight into what could be happening here? Other steps to try and rectify the problem? The database itself does not appear to have any issues that I can see.

Best,

Colin