Hi Bob: On Sat, Jul 5, 2014 at 3:34 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote: > Though currently I am thinking to address the problem of generating deltas > at the application level rather than to use postgresql features which are > largely optimized for a slightly different set of circumstances and > requirements. Yeah, special purpose can always be more efficient. Specially when you can tune for your special circunstances, like non full replication, and replicating only once a day, and you probably can minimizae the changesets knowing data dependencies. > Impressive what can be done witha 2400 baud modem when you set your mind to > it. Fortunately this days are mostly behind us :-) Not so behind. Then you did that kind of thing for 100Mb data sets, now you can use the same algorithms for 1Tb data sets. Line speed have gone up more than data set sizes, but this advantage has been eaten by customer delay expectations ( in my case, now they want at most 30 min delay, not 1440 ). Not knowing your exact data structure or size I cannot recommend anything, but I've been abble to make really good deltas using the same algorithms that were used on tapes, dump the table, sort it, diff with last days, the throughput of current machines is amazing, you can easily achieve full disk speed. Regards. Francisco Olarte.
