Am 28.11.2016 um 17:43 schrieb Daniel Verite:
Thomas Güttler wrote:
Up to now we use rsync (via rsnapshot) to backup our data.
But it takes longer and longer for rsync to detect
the changes. Rsync checks many files. But daily only
very few files really change. More than 99.9% don't.
lsyncd+rsync has worked nicely for me on Linux in such cases,
as opposed to rsync alone which is indeed very slow with large
trees. Check out https://github.com/axkibe/lsyncd
We reached the limit where inotify does not work any more. At least
that's what a team mate said. There are too many directories :-(
AFAIK lsyncd uses inotify.
If you think of using Postgres large objects, be aware that they
are stored in a single table (pg_largeobject), sliced
as rows of 1/4 block in size each (typically 2048 bytes).
2.3 TB in a single database would mean more than 1.2 billion
rows in that table, and as a system table it can't be partitioned
or moved to another tablespace.
OTOH with large objects, files can be stored and retrieved easily
between client and server with efficient built-in functions at both ends.
In particular, they don't need the binary<->text conversions or
large memory allocations mentioned by Chris Travers upthread,
that may happen when writing your own methods with bytea columns.
But for the amount of data you have, the monolithic pg_largeobject
would likely be problematic.
Ideally there should be an extension implementing something like
DATALINK (SQL99), with external storage. I wonder if an extension
could provide custom WAL records replicating content changes to the
external storage of a standby. That would be awesome.
Thank you for the hints. There is some info about DATALINK here:
https://wiki.postgresql.org/wiki/DATALINK
Regards,
Thomas Güttler
--
Thomas Guettler http://www.thomas-guettler.de/
--
Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
