Re: Hard link backup strategy

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Michael Monnerie <michael.monnerie@xxxxxxxxxxxxxxxxxxx> wrote:
> On Donnerstag 26 März 2009 Kevin Grittner wrote:
>> (1)  Our archive script copies WAL files to a directory on the
>> database server, using cp to a one directory followed by mv to
>> another (to prevent partial files from being processed).
> 
> What "partial files" do you speak about? I'd like to know, as we
> soon start doing WAL copies.
 
I don't want anything to attempt to process a WAL file while it is in
the process of being copied.  By copying to a separate directory on
the same mount point and moving it, once complete, to the location
where other software is looking for WAL files, we avoid that problem.
 
> You mean you make the hard link both on client and server, and 
> afterwards transfer with rsync? That only works for postgresql 
> <8.3, right? As I understand, 8.3 will reuse the space from updated 
> tuples and therefore lots of changes in-between will be done. 
 
PostgreSQL version has nothing to do with it.  PostgreSQL segments a
table into 1GB files.  Our largest tables are are either "insert only"
or rarely have updates or deletes, so many of these table segment
files are unchanged from one base backup to the next.  What we're
trying to accomplish is to use Linux hard links to create directory
entries from multiple base backups which point to the same file body.
Do a web search for "linux hard links" if this doesn't make sense to
you.
 
The benefit of rsync is that, when you use a daemon as we do, it won't
send unchanged portions of a file over the wire.  In preliminary tests
our largest county only would only send about 10% of the full set of
data if we used rsync on a snapshot of the original data directory,
without using cpio or gzip first.  I'm hoping that if we use the warm
standby image as the base, rather than the previous week's backup, it
will be a fraction of that.
 
Basically, the hard links will be used to conserve disk space; rsync
will be used to conserve network bandwidth.
 
-Kevin

-- 
Sent via pgsql-admin mailing list (pgsql-admin@xxxxxxxxxxxxxx)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux