I have managed to split the 365GB file into 2GB chunks with the help of 'split' unix utility in mingw shell like so
split -C 2GB
ways.txt
Then I imported the files into a clean database with the help of the following cmd command
for /f %f in ('dir /b') do psql -U postgres -w -d osm -t -c "set client_encoding TO 'UTF8'; copy ways from 'D:\ways\%f';"
The operation took ~3.5 hour which is the same as v10!
Prior to that I set 'parallel_leader_participation = on' and 'synchronous_commit = off' in the config file and restarted the server.
Then I logged into the psql interactive terminal and ran
ALTER TABLE ONLY ways ADD CONSTRAINT pk_ways PRIMARY KEY (id);
It took 1h 10m which is 30m faster than with the default settings (after 'type' commad if it really matters) but still 3 times slower than in v10.
Regards
пн, 2 дек. 2019 г. в 12:04, Laurenz Albe <laurenz.albe@xxxxxxxxxxx>:
On Sat, 2019-11-30 at 22:47 +0300, Eugene Podshivalov wrote:
> It sounds strange but the "type" is indeed impacting the overall
> performance somehow.
> I've just tried to execute the following sequence of commands on a
> fresh new database with PostreSQL v10 and both the copy and primary
> key commands performed as slow as in v11 and 12.
>
> SET synchronous_commit TO OFF;
> SET client_encoding TO 'UTF8';
> COPY ways FROM program 'cmd /c "type D:\ways.txt"';
> ALTER TABLE ONLY ways ADD CONSTRAINT pk_ways PRIMARY KEY (id);
>
> Regards
>
> пт, 29 нояб. 2019 г. в 15:22, Eugene Podshivalov <yaugenka@xxxxxxxxx>:
> > Laurenz,
> > There is no way to run copy without the "type" on v11. See this thread
> > https://www.postgresql.org/message-id/CAKHmqNCfTMM6%3DPqc6RUMEQ_2BPfo5KGGG-0fzRXZCVooo%3DwdNA%40mail.gmail.com
> >
> > My machine is running on NVMe disks, so the I/O subsystem very strong.
> > The 100% overload is not constant but periodical, as if there are some
> > kind of dumps for recovery performed in the background.
Is it an option to split the file into parts of less than 2GB in size?
Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
