Postgres PANIC when it could not open file in pg_logical/snapshots directory

[Mike Yeap <wkk1020@xxxxxxxxx>]

Hi all,

I have a Postgres version 11.11 configured with both physical replication slots (for repmgr) as well as some logical replication slots (for AWS Database Migration Service (DMS)). This morning, the server went panic with the following messages found in the log file:

2021-06-22 04:56:35.314 +08 [PID=19457 application="[unknown]" user_name=dms database=** host(port)=**(48360)] PANIC:  could not open file "pg_logical/snapshots/969-FD606138.snap": Operation not permitted

2021-06-22 04:56:35.317 +08 [PID=1752 application="" user_name= database= host(port)=] LOG:  server process (PID 19457) was terminated by signal 6: Aborted

2021-06-22 04:56:35.317 +08 [PID=1752 application="" user_name= database= host(port)=] LOG:  terminating any other active server processes

The PG server then terminates all existing PG processes.

The process with 19457 is from one of the DMS replication tasks, I have no clue why it suddenly couldn't open a snapshot file. I checked the server load and file systems and didn't find anything unusual at that time.

Appreciate if you can give me some guidance on troubleshooting this issue

Thanks

Regards,

Mike Yeap