Re: optimizing import of large CSV file into partitioned table?

[Rick Casey <caseyrick@xxxxxxxxx>]

Thanks Dim; I was not aware of pgloader. This, and the other suggestions, have helped a lot; thanks everyone.

--rick

On Mon, Mar 29, 2010 at 7:41 AM, Dimitri Fontaine <dfontaine@xxxxxxxxxxxx> wrote:

Rick Casey <caseyrick@xxxxxxxxx> writes:

> So, I am wondering if there is any to optimize this process? I have been using Postgres for several years, but have never had to partition or optimize it for files
> of this size until now. 
> Any comments or suggestions would be most welcomed from this excellent forum.

The pgloader tool will import your data as batches of N lines, you get
to say how many lines you want to consider in each transaction. Plus,
you can have more than one python thread importing your big file, either
sharing one writer and having the other threads doing the parsing and
COPY, or having N independent threads doing the reading/parsing/COPY.

 http://pgloader.projects.postgresql.org/

Hope this helps,
--
dim

--
----------------------------------------------------------------------------
Rick Casey :: caseyrick@xxxxxxxxx :: 303.345.8893