On Wed, Nov 22, 2023 at 2:28 PM Tom Lane <tgl@xxxxxxxxxxxxx> wrote:
"Efrain J. Berdecia" <ejberdecia@xxxxxxxxx> writes:
> Thanks, the issue we've run into, which I guess could be really a setup issue, with running a COPY command while executing pg_restore, is that if we are restoring a large table (bigger than 500GB) our WAL directory can grow to be very large.
> I would think that if the pg_restore or COPY command was able to support a batch-size option, this should allow postgres to either archive or remove wal files and prevent having to re-size the WAL directory for a one time refresh operation.
> I'm trying to gage how feasible would be to start looking at contributing to add such a feature to either the COPY command or pg_restore.
Given the shortage of other complaints, I tend to agree with Adrian
that there's not likely to be much interest in adding complexity
to pg_restore (or COPY) to address this. You should probably look
harder at the idea that you have some configuration problem that's
triggering your WAL bloat. If COPY can run you out of WAL space,
then so could any future bulk insert or update.
What OP needs, I think, since I'd use it, too, is "pg_bulkload without the intrusive hacks and restrictions".
