Re: LARGE single-system Cyrus installs?

[Ian G Batten <ian.batten@xxxxxxxxxxxxxx>]

On 17 Nov 07, at 0909, Rob Mueller wrote:

This shouldn't really be a problem. Yes the whole file is locked for the 

duration of the write, however there should be only 1 fsync per 

"transaction", which is what would introduce any latency. The actual writes 

to the db file itself should be basically instant as the OS should just 

cache them.

One thing that's worth noting for ZFS-ites is that on ZFS, you can have multiple writer threads in a file simultaneously, which UFS can only do for directio under certain conditions I can't recall.  That's a win for overlapping transactions into a file-based database.   We're not hitting mailboxes.db remotely rapidly enough for this to be an issue, but I can imagine it being so for big shops.

In production releases of ZFS fsync() essentially triggers sync() (fixed in Solaris Next).  So if you anticipate a lot of writes (and hence fsync()s) to mailboxes.db then you don't want mailboxes.db in the same ZFS filesystem as things with lots of un-sync'd writes going on.    I've broken up /var/imap for ease of taking and rolling back snapshots, but it has the handy side-effect of isolating delivery.db and mailboxes.db from all the metadata partitions.

In my darker moments, by the way, I'm tempted to put deliver.db into tmpfs.  For planned reboot I could copy it somewhere stable, and I could periodically dump it out to disk.  But if I lost it, the consequences aren't serious, and it's most of the write load through that particular filesystem.

ian

mailhost-new# zfs list -t filesystem | grep imap; df /var/imap/proc

pool1/mailhost-space/imap              1.34G  24.6G   346M  /var/imap

pool1/mailhost-space/imap-seen          105M  24.6G  22.4M  /var/imap/user

pool1/mailhost-space/meta-partition-1  2.48G  24.6G   972M  /var/imap/meta-partition-1

pool1/mailhost-space/meta-partition-2  12.4G  24.6G  4.82G  /var/imap/meta-partition-2

pool1/mailhost-space/meta-partition-3  4.86G  24.6G  1.60G  /var/imap/meta-partition-3

pool1/mailhost-space/meta-partition-7  5.60G  24.6G  1.41G  /var/imap/meta-partition-7

pool1/mailhost-space/meta-partition-8  14.0G  24.6G  5.39G  /var/imap/meta-partition-8

pool1/mailhost-space/meta-partition-9  1.08G  24.6G   415M  /var/imap/meta-partition-9

pool1/mailhost-space/sieve             5.26M  24.6G  1.62M  /var/imap/sieve

/var/imap/proc     (swap              ):  514496 blocks  2356285 files

mailhost-new# 

Still, you have a point that mailboxes.db is a global point of contention, 

and it is access a lot, so blocking all processes on it for a write could be 

an issue.

Which makes me even more glad that we've split up our servers into lots of 

small cyrus instances, even less points of contention...

Rob

----

Cyrus Home Page: http://cyrusimap.web.cmu.edu/

Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki

List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html