On 7/3/2014 10:13 PM, Bosco Rama wrote:
Is the issue with S3 or just transfer time? I would expect that 'rsync' with the '--partial' option (or -P if you want progress info too) may help there.
Don't know if rsync and S3 work together or what that would mean, but it's not an issue I'm suffering now. I do think they may now have a multipart upload with s3cmd (which I use), though that also wasn't available when we first built our scripts.
I suspect nothing is really helping here and I'm mostly limited by disk I/O, but not sure why the pg_dump is so much slower than pg_restore as they are all on the same disks. I say this because even with pg_dump -Z0 | gpg -z 0 and gzip removed entirely and no --oids on pg_dump, there's no effective difference in overall speed. While I can see all of those processes vying for resources via 'top -c', the throughput remains much the same.
