Re: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3

[Raj kumar <rajkumar820999@xxxxxxxxx>]

Correction ** - To enable logical replication of a AWS-RDS PostgreSQL source database, I followed the below steps.

On Sun, Dec 12, 2021 at 4:18 PM Raj kumar <rajkumar820999@xxxxxxxxx> wrote:

AWS DMS supports change data capture (CDC) using logical replication. I'm using PostgreSQL V12 as the DMS Source and migrating data to S3 DataLake through DMS.

After creating replication slot for pglogical and enabling CDC from Postgresql RDS to S3, the Storage space consumed heavily and increased by 1TB in a day and filled up the storage completely. What could be the reason for this? How can I avoid this? Please advice.

To enable logical replication of a self-managed PostgreSQL source database, I followed the below steps.

1. Set wal_level = logical.
2. Set max_replication_slots to a value greater than 1.
3. Set the max_replication_slots value to 10
4. Set max_wal_senders to a value greater than 10.
5. Set wal_sender_timeout =0.

GRANT USAGE on SCHEMA pglogical to DATALAKE_USER;

GRANT ALL on ALL TABLES in SCHEMA pglogical to DATALAKE_USER;

SELECT pglogical.create_node(node_name := 'postgresdb',dsn := 'host=postgresinstancedb.ap-south-1.rds.amazonaws.com port=5432 dbname=masterpoc');

select pglogical.create_replication_set('pgdms_replication_slot_datalake', true, false, false, true);

Thanks & Regards,
Raj Kumar Narendiran.