On Mon, Mar 11, 2024 at 12:33 AM Avi Weinberg <AviW@xxxxxxxxx> wrote: > > Hi Experts, > > > > Your input is most welcome! > > > > We are using Postgres 13 (and plan to upgrade to 15 soon). We have logical replication with about 40 servers subscribing to one publisher. 40 Walsender processes are running on the publisher server. When we insert a row into a table holding binary data the walsender RAM usage increases by 500MB although the row binary data is only 80MB. We see this increase in all walsender processes. At some point we got OOM and the process was killed. This sounds like similar reports[1][2] we got before. Were there any long-running transactions at that time when the 80MB data change was made? And is it accessible to the core dump of the walsender process who was killed due to OOM? > Why does the walsender increases by 500MB when the data change was only 80MB > Is some of the 500MB increase due to shared memory or each walsender has its own 500MB increase. I assume that if it was only in shared memory we would not have gotten OOM… > Why when logical_decoding_work_mem = 64MB the RAM is 15 times that size? Shouldn't any additional space be used from disk and not RAM? > Will adding streaming = on to publication "PUBLICATION pub WITH (streaming = on)" can alleviate the issue? > Are there configuration options that can resolve the RAM issue. It can be also in version 15 since we plan to upgrade soon. If you're facing a similar issue I shared above, temporarily setting logical_decoding_work_mem a *lower* value could alleviate the situation. Setting a lower value would lead to more evictions in logical decoding, and it probably can avoid using much memory and OOM (note that the logical decoding gets slower instead). I think there is a memory accounting issue in logical decoding, which could end up using memory much more than the logical_decoding_work_mem limit in some scenarios. This issue is not fixed yet, and setting "streaming = on" doesn't help. Regards, [1] https://www.postgresql.org/message-id/CAMnUB3oYugXCBLSkih%2BqNsWQPciEwos6g_AMbnz_peNoxfHwyw%40mail.gmail.com [2] https://www.postgresql.org/message-id/17974-f8c9d353a62f414d%40postgresql.org -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
