(Sorry, that's a lot of big words. I'll try explaining what I want to do.)
On my LAN I have a Cyrus IMAP server (3.0.11), and a dedicated Cyrus
backup server (patched with Ellie's shared-mailbox and 64-bit fseek
fixes). These are connected by a nice fat link so backups happen fast
and often. A scheduled compaction occurs each morning thanks to an
event in cyrus.conf.
I now want to back up the backups to an off-site server over a much
slower link. The off-site server doesn't speak the Cyrus sync
protocol. What it does do well is block-level backups: if only a part
of a file has changed, only that part needs to be transferred over the
slow link. [I haven't decided whether my technology will be the rsync
--checksum protocol, or Synology NAS XFS replication, or Microsoft
Server VFS snapshots. They all do block-level backups well.]
Since Cyrus backup files are append-only, they should behave well with
block-level backups. But—correct me if I'm wrong—compaction is going to
ruin my day because a reduction in the size of chunk (say) 5 moves the
start offset of chunk 6 (and so on). Even if chunk 6 doesn't change
it'll have to be retransmitted in its entirety.
Right now I've set backup_compact_minsize and backup_compact_maxsize to
zero but I'm not sure if even that is sufficient to prevent chunk
offsets moving. Perhaps I need to disable the compaction event in
cyrus.conf entirely.
I really want compaction, though, or else my backups are going to get
very, very big.
Which leads me to my idea. What if compaction could be friendlier
towards block-level backups, by deliberately avoiding changing chunk
offsets in the backup file, even if that means gaps of unused bytes when
(the aforementioned) chunk 5 shrinks? It won't always work out, for
instance when a chunk grows in size. A compromise would need to be
struck between keeping chunk offsets fixed and wasted fragmented space
between chunks as they shrink.
I haven't collected enough data to know if I am making the right
assumptions about how chunk size evolves over time and how effective
compaction is at removing cruft from a backup file. Has anyone thought
about doing something like this with Cyrus backups?
--
Deborah Pickett
System Administrator
Polyfoam Australia Pty Ltd
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
