Ok - we're doing our GUID upgrades across the board now. Here's
the process we're using:
a) wrote a tool that tied a "DB_File" called "cyrus.sha1s" in each
meta directory on the replicas, parsed the index files looking
for records with nothing but zeros in the last 16 characters of
the GUID and calculated the sha1 on each of them.
This took about 5 days to finish running, but makes the next
part a lot quicker!
b) wrote a daemon which runs on each host and allows the following
4 commands:
*) LOCK user.name.URI%20Escaped.MailboxName
- lock cyrus.header, cyrus.index, cyrus.expunge in that order
using fcntl (our cyrus is build with it).
Also if cyrus.sha1s is missing, attempt to fetch it from the
replica (but it's OK if this fails, just means all old GUIDs
will cause a re-calculation)
*) UPGRADE
- parse each of cyrus.index and cyrus.expunge. If any old-style
GUIDs are found, then looks first in cyrus.sha1s and finally
just re-calculates from the underlying message files.
- if any index records need new GUIDs or the old index has not
yet been upgraded to version 10, stream the index file thorugh
a Cyrus::IndexFile->stream_copy, altering the necessary GUIDs
and forcing the output format to version 10 (this module can
also be used to downgrade if we ever need to!)
- leave the new file in cyrus.$item.NEW, but mark internally
that the file has been upgraded.
*) ROLLBACK
- if any file has been upgraded, unlink() the .NEW file.
- unlock expunge, index, header (in that order)
*) COMMIT
- if any file has been upgraded, rename() the .NEW to the
base filename.
- unlock expunge, index, header (in that order)
c) wrote a controller script which reads the mailbox listing from the
master and opens connections to both the master and replica slotd,
sending the following commands:
1) master LOCK mailbox (or die)
2) replica LOCK mailbox (or master ROLLBACK; die)
3) master UPGRADE (or replica ROLLBACK; master ROLLBACK; die)
4) replica UPGRADE (or replica ROLLBACK; master ROLLBACK; die)
5) replica COMMIT (or replica ROLLBACK; master ROLLBACK; die)
6) master COMMIT (or master ROLLBACK; die NOISILY!!!)
The only danger point is (6), where you could wind up with an
upgraded replica without the associated upgraded master. You
can go ahead and fix them by hand though, assuming you read the
NOISILY bit.
d) I think the slashdot crowd would put "Profit!!!" here. With
only a short lock time on each index (most sha1s precalculated)
and no need to multi-rewrite any index file, this will run much
faster than the alternatives. I guess I should go clean up the
cyrus.sha1s files once it's all finished.
Bron.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
