Search Postgresql Archives

Re: How can you find out what point logical replication is at? -- or weird, slow, infinite loop

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This has gotten a bit weirder.

The replication is getting up to a specific LSN.  

Then it moves the sent_lsn to a number less than that, for a WAL file about an hour or more earlier and slowly walks forward again till it hit the same highwater mark.

So the replication seems to be stuck in some kind of a slow infinite loop.

Corrupted WAL file?  Any other ideas of what to look for?

TIA,

John

On Aug 3, 2020, at 10:38 AM, John Ashmead <john.ashmead@xxxxxxxxxxxxxxxxxxx> wrote:

I have logical replication setup from a factory in Zhuhai China to a data warehouse in New Jersey. We are using postgresql 10.13 on both sides, on Redhat Linux 7.6.

The logical replication has been in “catchup” mode for several days now, stuck at a specific LSN (9EF/89ADF7E0). The slave side seems to be chugging along, generating lots of WAL files — but not actually getting any new records in. 

The logical replication is being used to move some fairly large files:  averaging about 1 MB but with a max up to about 250 MB. (I think I want to fix that longer term, but that’s not a quick fix.)

My working hypothesis is that we are stuck on a long transaction: that we can’t get some abnormally large blob over before we drop the line.  In this case fixing the connection should fix the problem.

Is there a way I can see what is going on?  The slave is working hard, but what is it working on?

And are there any suggestions on how to handle this?

I could restart the logical replication with “copy_data = false”, then fill in the holes by hand.  But I would rather not!

Thanks in advance!

John

PS. Output of pg_stat_replication & pg_stat_subscription on master & slave respectively.  (Some proprietary information X’d out)

select * from pg_stat_replication;
-[ RECORD 1 ]----+------------------------------
pid              | 42451
usesysid         | 10
usename          | postgres
application_name | china_to_nj_sub
client_addr      | XXX.XXX.XXX.XXX
client_hostname  | 
client_port      | 54300
backend_start    | 2020-08-03 09:07:07.257454-04
backend_xmin     | 16574498
state            | catchup
sent_lsn         | 9EF/89ADF7E0
write_lsn        | 9EF/89ADF7E0
flush_lsn        | 9EF/89ADF7E0
replay_lsn       | 9EF/89ADF7E0
write_lag        | 
flush_lag        | 
replay_lag       | 
sync_priority    | 0
sync_state       | async

 select * from pg_stat_subscription;
-[ RECORD 1 ]---------+------------------------------
subid                 | 7222699
subname               | china_to_nj_sub
pid                   | 14764
relid                 | 
received_lsn          | 9EF/89ADF7E0
last_msg_send_time    | 2020-08-03 10:15:48.644575-04
last_msg_receipt_time | 2020-08-03 10:14:57.247993-04
latest_end_lsn        | 9EF/89ADF7E0
latest_end_time       | 2020-08-03 09:30:57.974223-04


John Ashmead
139 Montrose Avenue
Rosemont, PA, 19010-1508
(610) 527 9560 
mobile (610) 247 2323







John Ashmead
139 Montrose Avenue
Rosemont, PA, 19010-1508
(610) 527 9560 
mobile (610) 247 2323






<<attachment: smime.p7s>>


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Postgresql Jobs]     [Postgresql Admin]     [Postgresql Performance]     [Linux Clusters]     [PHP Home]     [PHP on Windows]     [Kernel Newbies]     [PHP Classes]     [PHP Books]     [PHP Databases]     [Postgresql & PHP]     [Yosemite]

  Powered by Linux