On Mon, 28 Nov 2016 15:28:28 +0100 Thomas Güttler <guettliml@xxxxxxxxxxxxxxxxxx> wrote: > Hi, > > Up to now, we don't store files in PostgreSQL. > > I was told, that you must not do this .... But this was 20 years ago. > > > I have 2.3TBytes of files. File count is 17M > > Up to now we use rsync (via rsnapshot) to backup our data. > > But it takes longer and longer for rsync to detect > the changes. Rsync checks many files. But daily only > very few files really change. More than 99.9% don't. > > Since we already store our structured data in postgres, I think > about storing the files in PostgreSQL, too. > > What is the current state of the art? > > Is it feasible to store file in PostgreSQL? Yes and no, it's another level of indirection, slower than pure filesystem solution. Rsync checks last read/access time, last write/modification time and file hash before copying it. If no one of those metadata change, rsync don't copy it. File hash must be recalculated if access time and modification time change. > Are there already projects which use PostgreSQL as storage backend? > > I have the hope, that it would be easier to backup only the files > which changed. Rsync tries to backup only the files that changed. There are other tools like cpdup, don't know if it's ported to linux, It's similar to rsync. You can use a p2p system, unlike ftp, rsync, etc they store a full Tree Hash (Tiger Tree Hash often) of file content to allow multi peer to peer copy. > Regards, > Thomas Güttler > > > Related question at rsnapshot mailing list: > https://sourceforge.net/p/rsnapshot/mailman/rsnapshot-discuss/thread/57A1A2F3.5090409@xxxxxxxxxxxxxxxxxx/ > -- > Thomas Guettler http://www.thomas-guettler.de/ > --- --- Eduardo Morras <emorrasg@xxxxxxxx> -- Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
