Re: LARGE single-system Cyrus installs?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 20 Nov 07, at 1756, David Lang wrote:


however a fsync on a journaled filesystem just means the data needs to be 
written to the journal, it doesn't mean that the journal needs to be flushed to 
disk.

on ext3 if you have data="" then your data is in the journal as well and 
all that the system needs to do on a fsync is to write things to the journal (a 
nice sequential write),

Assuming the journal is on a distinct device and the distinct device can take the load.  It isn't on ZFS, although work is in progress.  One of the many benefits of the sadly underrated Solaris Disksuite product was the metatrans devices, which at least permitted metadata updates to go to a distinct device.  When the UFS logging code went into core Solaris (the ON integration) that facility was dropped, sadly.  My Pillar NFS server does data logging to distinct disk groups, but mostly --- like such boxes tend to do --- relies on 12GB of RAM and a battery.  A sequential write is only of benefit if the head is in the right place and the platter is at the right rotational position and the write is well-matched to the transfer rate of the spindle: if the spindle is doing large sequential writes while also servicing reads and writes elsewhere, or can't keep up with writing tracks flat out, the problems increase.

for cyrus you should have the same sort of requirements that you would have for 
a database server, including the fact that without a battery-backed disk cache 
(or solid state drive) to handle your updates, you end up being throttled by 
your disk rotation rate (you can only do a single fsync write per rotation, and 
that good only if you don't have to seek), RAID 5/6 arrays are even worse, as 
almost all systems will require a read of the entire stripe before writing a 
single block (and it's parity block) back out, and since the stripe is 
frequently larger then the OS readahead, the OS throws much of the data away 
immediatly.

if we can identify the files that are the bottlenecks it would be very 
interesting to see the result of puttng them on a solid-state drive.

I've split the meta-data out into separate partitions.  The meta data is stored in ZFS filesystems in a pool which is a RAID 0+1 4 disk group with SAS drives, the message data is coming out of the lowest QoS on my Pillar.  A ten second fsstat on VM operations shows that by request (this measures filesystem activity, not the implied disk activity) it's the meta partitions taking the pounding (ten second sample):

  map addmap delmap getpag putpag pagio
    0      0      0     45      0     0 /var/imap
   11     11     11     17      0     0 /var/imap/meta-partition-1
  290    290    290    463      5     0 /var/imap/meta-partition-2
  139    139    139    183      3     0 /var/imap/meta-partition-3
   66     66     66    106     10     0 /var/imap/meta-partition-7
  347    347    342    454     16     0 /var/imap/meta-partition-8
   57     57     57     65      5     0 /var/imap/meta-partition-9
    4      4      8      4      0     0 /var/imap/partition-1
   11     11     22     14      0     0 /var/imap/partition-2
    1      1      2      1      0     0 /var/imap/partition-3
    6      6     12     49     10     0 /var/imap/partition-7
   15     15     28    457      0     0 /var/imap/partition-8
    1      1      2      2      0     0 /var/imap/partition-9

Similarly, by non-VM operation:

 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
    0     0     0 2.26K     0  6.15K     0     0     0    45 1.22K /var/imap
    0     0     0   356     0    707     0     0     0     6 3.03K /var/imap/meta-partition-1
    3     0     3   596     0    902     0     6  135K    90  305K /var/imap/meta-partition-2
    0     0     0   621     0  1.08K     0     0     0     3 1.51K /var/imap/meta-partition-3
    3     0     3 1.04K     0  1.70K     0     6  149K    36  650K /var/imap/meta-partition-7
    0     0     0 2.28K     0  4.24K     0     0     0     7 1.87K /var/imap/meta-partition-8
    0     0     0    18     0     32     0     0     0     2   176 /var/imap/meta-partition-9
    2     2     2    22     0     30     0     1 2.37K     2 7.13K /var/imap/partition-1
    3     4    12    84     0    157     0     1   677     3 7.51K /var/imap/partition-2
    1     1     1 1.27K     0  2.16K     0     0     0     1 3.75K /var/imap/partition-3
    2     2     4    35     0     56     0     1 3.97K    36  279K /var/imap/partition-7
    1     2     1   256     0    514     0     0     0     1 3.75K /var/imap/partition-8
    0     0     0     0     0      0     0     0     0     0     0 /var/imap/partition-9


And looking at the real IO load, ten seconds of zpool (for the meta data and /var/imap_

                 capacity     operations    bandwidth
pool           used  avail   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
pool1         51.6G  26.4G      0    142  54.3K  1001K
  mirror      25.8G  13.2G      0     68  38.4K   471K
    c0t0d0s4      -      -      0     36  44.7K   471K
    c0t1d0s4      -      -      0     36      0   471K
  mirror      25.8G  13.2G      0     73  15.9K   530K
    c0t2d0s4      -      -      0     40  28.4K   531K
    c0t3d0s4      -      -      0     39  6.39K   531K
------------  -----  -----  -----  -----  -----  -----

is very different to ten seconds of sar for the NFS:

09:46:34   device        %busy   avque   r+w/s  blks/s  avwait  avserv

[...]
           nfs73             1     0.0       3     173     0.0     4.2
           nfs86             3     0.1      12     673     0.0     6.5
           nfs87             0     0.0       0       0     0.0     0.0
           nfs89             0     0.0       0       0     0.0     0.0
           nfs96             0     0.0       0       0     0.0     1.8
           nfs101            1     0.0       1      25     0.0     8.0
           nfs102            0     0.0       0       4     0.0     9.4

The machine has a _lot_ of memory (32GB) so it's likely that all mail that is delivered and then read within ten minutes never gets read back from the message store: the NFS load is almost entirely write as seen from the server.

ian
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

[Index of Archives]     [Cyrus SASL]     [Squirrel Mail]     [Asterisk PBX]     [Video For Linux]     [Photo]     [Yosemite News]     [gtk]     [KDE]     [Gimp on Windows]     [Steve's Art]

  Powered by Linux