Hi. The doc states:
From https://www.postgresql.org/docs/current/libpq-copy.html:
> Data is always returned one data row at a time;
> if only a partial row is available, it is not returned
I'm doing a COPY TO STDOUT WITH (FORMAT BINARY).
The record format allows up to int16_t columns. (16K or 32K, doesn't matter here).
Each column can be TOAST'd up to 1GB.
From https://www.postgresql.org/docs/current/libpq-copy.html:
> Data is always returned one data row at a time;
> if only a partial row is available, it is not returned
I'm doing a COPY TO STDOUT WITH (FORMAT BINARY).
The record format allows up to int16_t columns. (16K or 32K, doesn't matter here).
Each column can be TOAST'd up to 1GB.
And as far as I know, COPY de-toasts values, like SELECT.
So "in theory", the row can be ginormous, and blow out RAM.
And even if it doesn't, on a 1GB RAM machine for example,
you'd have to wait a looonng time for that one row.
Even if you have only 1 or 2 large values (text or bytea),
you don't want to wait for 1GB or 2GB of data doing nothing,
but instead "stream" the partial row content to its final destination
(be it processing, write to disk, whatever) concurrently.
So, is the current documented behavior a design choice (limitation) of libpq,
or something more profound from the v3 PROTOCOL?
And is libpq ever going to offer an alternative that's more "streaming" friendly, in case of big rows?
As far as I know, on the write side (COPY FROM STDIN BINARY),
there is no requirement at the libpq client API level to write entire rows at a time,
one can write any partial "row-unaligned" chunk of bytes (properly encoded).
And I assume libpq does not buffer those bytes, waiting for entire "aligned" rows,
before sending them output to the server via the socket. Right?
I understand the convenience of getting one-row-at-the-time from PQgetCopyData(),
but that's less than ideal in the general case. Thus my questions above.
Thanks, --DD
