Hi Ron,
Uploads to a remote server?
— Yes, to S3
— Yes, to S3
Does wal-g compress files before sending them across the wire? By how much? Are you CPU or IO bound by having to compress that much data?
— Yes, we use default compression i.e lz4. There is no pressure on resources but the number of wal files that are processed in parallel, if we increase the streams it uses up a lot of memory.
Thanks,
Nikhil
On Sun, 6 Aug 2023 at 13:51, Ron <ronljohnsonjr@xxxxxxxxx> wrote:
On 8/6/23 02:43, Nikhil Shetty wrote:
Hi Team,
I would like to know which backup&restore tools will be better for scenarios where the database is generating around 400 WALs per minute.
If my math is correct, 400x 16MB WAL files per minute is 400*(16*2^20)/60*8 / 10^6 = 895 MBits per second. Plus overhead.
That's about 1Gbit/second. Definitely nothing to sneeze at.
We are using wal-g but it is not able to keep pace with the wal generation. We increased the upload streams to 256 but no luck
Uploads to a remote server?
Does wal-g compress files before sending them across the wire? By how much? Are you CPU or IO bound by having to compress that much data?
--
Born in Arizona, moved to Babylonia.
