> how did you determine that it is done every 500 rows? this is the The import program pages the import table - it is currently set at 500 rows per page. With each page, I run an ANALYZE. > default autovacuum paramater. if you followed my earlier > recommendations, you are aware that autovacuum (which also analyzes) > is not running during bulk inserts, right? It's intuitivly obvious, but I can't do bulk inserts. It's just not the nature of what we are doing with the data. > imo, best way to do big data import/conversion is to: > 1. turn off all extra features, like stats, logs, etc done > 2. use copy interface to load data into scratch tables with probably > all text fields done > 3. analyze (just once) I think this doesn't apply in our case, because we aren't doing bulk inserts. > 4. use big queries to transform, normalize, etc This is currently being done programmatically. The nature of what we're doing is suited for imperitive, navigational logic rather than declarative, data set logic; just the opposite of what SQL likes, I know! If there's some way to replace thousands of lines of analysis and decision trees with ultrafast queries - great... > important feature of analyze is to tell the planner approx. how big > the tables are. But the tables grow as the process progresses - would you not want the server to re-evaluate its strategy periodically? Carlo > > merlin > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org >
