On Tue, Nov 28, 2017 at 11:19 AM, Rob Sargent <robjsargent@xxxxxxxxx> wrote: > >> On Nov 28, 2017, at 10:17 AM, Ted Toth <txtoth@xxxxxxxxx> wrote: >> >> I'm writing a migration utility to move data from non-rdbms data >> source to a postgres db. Currently I'm generating SQL INSERT >> statements involving 6 related tables for each 'thing'. With 100k or >> more 'things' to migrate I'm generating a lot of statements and when I >> try to import using psql postgres fails with 'out of memory' when >> running on a Linux VM with 4G of memory. If I break into smaller >> chunks say ~50K statements then thde import succeeds. I can change my >> migration utility to generate multiple files each with a limited >> number of INSERTs to get around this issue but maybe there's >> another/better way? >> >> Ted >> > what tools / languages ate you using? I'm using python to read binary source files and create the text files contains the SQL. Them I'm running psql -f <file containing SQL>.
