Hi there,
I am trying to build an application which reads messages off of a replication slot and does some processing with them. I want to guarantee that every event gets processed exactly once. Unfortunately, I've been unable to find a guide to the correct way to read this data. I've been experimenting and it seems to work, but I wanted to confirm a few things. Apologies if this is the wrong place for this kind of question, or if reading off a slot is described elsewhere (if so, I would appreciate being pointed to the right place).
I'm trying to use terminology from https://www.postgresql.org/docs/9.5/static/protocol-replication.html
My basic idea is:
* On startup, create a replication connection and do START CONNECTION with LSN 0/0. This seems to start off from the last "committed" LSN (last LSN sent back in a standby status update).
* Keep track of a "last committed" LSN, and periodically send back a Standby Status Update with the "written", "flushed", and "applied" LSNn all set to that value (I could not find documentation on the difference between the three, but setting all of them equal seems to work).
* When I see an XLogData message, process the data and then set the "last committed" LSN to the "starting point of the WAL data" LSN.
* When I see a Primary Keepalive Message, set "last committed" LSN to the "current end WAL" field, assuming all prior XLogDatas have been processed. If the "should reply" byte is set, send back a Standby Status Update immediately
Is this the way that this feature was intended to be used?
In particular, I have two concerns with this approach:
1) On the XLogData, should I be using "starting point of WAL data" or "current end of WAL"? In my testing, they seem to always be the same, so I'm worried there's some case I haven't handled.
2) I'm not sure if committing the end WAL LSNs on the Keepalive messages is safe. Is it possible to receive a Keepalive message, followed by an XLogData message, such that the Keepalive's LSN is higher than the XLogData's? This could cause problems, because after committing the Keepalive's LSN, if my application needed to restart, when it comes back up, we might miss the XLogData, since we committed a higher LSN.
Thanks, and I appreciate any pointers you can provide!
-Matt
