Why "copy ... from stdio" does not return immediately when reading invalid data?

[Nicolas Grilly <nicolas@xxxxxxxxxxxxxxxx>]

Hello,

I am importing gigabytes of data into PostgreSQL, and I don't want to wait 10 minutes just to discover an error in the 10th line of my input file.

I tried the command "\copy ... from stdio" in psql and it looks like psql has to read the entire input before returning a potential error, even if the invalid value is in one of the first rows.

Is it a limitation of PostgreSQL protocol, of the library lipq, or of the tool psql?

Is the copy protocol (aka PQputCopyData and PQputCopyEnd) designed to send gigabytes of data with just one "copy ... from stdio" query,Âand is there a way to be notified of a potential error beforeÂcalling PQputCopyEnd?ÂOr do I have to send my data in small chunks (for example batch of 10000 rows),Âissue a PQputCopyEnd, check for errors, and continue with the next chunk?

Thanks for your help and advice.

Regards,

Nicolas Grilly