The thought: - Load the big chunk of data into a new table - Generate some minimal set of indices on the new table - Generate four queries that compare old to new: q1 - See which tuples are unchanged from yesterday to today q2 - See which tuples have been deleted from yesterday to today q3 - See which tuples have been added q4 - See which tuples have been modified If the "unchanged" set is extremely large, then you might see benefit to doing updates based on deleting the rows indicated by q2, inserting rows based on q3, and updating based on q4. In principle, computing and applying those 4 queries might be quicker than rebuilding from scratch. In principle, applying q2, then q4, then vacuuming, then q3, ought to be "optimal."
This looks like an interesting idea, and I'm going to take a look at how feasible it'll be to impletement. I may be able to combine this with Mr. Wagner's idea to make a much more efficient system overall. It's going to be a pretty big programming task, but I've a feeling this summarizer thing may just need to be re-written with a smarter system like this to get something faster.
Thanks! Steve
