Just a little note, for those who have perhaps the same problem.We saw performance problems after we switched from a Linux installation to a Solaris 10 cluster connected to our SAN (using scsi_vhci and 2 Qlogic Controllers).
Problems arose when real load came to the machine, despite having tested it with some load simulation scenarios...
Looking at the attached chart[0], you will notice up to 10 seconds time to select a mailbox, far too long.
The problem was ZFS - ZFS is perfect for rubbish, cheap disks with ugly firmwares. So if your storage is "too good" it makes things go bad.
First: The file prefetch algorithm does not seem to be very good for 20 million mail files and 300,000 cyrus meta files... ;-)
So first step was (at 11am in the chart) to disable this prefetching routine. [1]
This cut off 4 seconds from requests, but still 10 second-timeouts were seen.
Next step was to disable the zfs cache flush. As said, ZFS is good for rubbish disks, so every 5 seconds it instructs the SATA or SCSI drives to flush their internal ram to disk. This is good for cheap disks, but a no-go if you have a SAN RAID storage having 2 Gigabytes and more of RAM storage, buffered with a battery. In fact, our storage system really flushed all 5 second its complete RAM cache, you even saw it watching the "blinkenlights".
Plus, every fsync() call did the same... :(There is a trick: You can disable this in ZFS [2]. You see the result at 4pm (16:00 european notation in the chart). CHILDREN DO NOT TRY THIS AT HOME..... Only do this if you don't have any "real" physical disk storages attached to your system with zfs pools on them - otherwise you will lose data on power outages as the ram cache of your hard disk is not buffered by a battery.
Now the machine runs and handles all mail without noticeable delays. [0] http://priscilla.rz.uni-konstanz.de/mailserver/ [1] in /etc/system: set zfs:zfs_array_rd_sz=0 on a live system using mdb -kw: zfs_array_rd_sz/Z0x0 [2] in /etc/system: set zfs:zfs_nocacheflush=1 on a live system using mdb -kw: zfs_nocacheflush/W0t1
Attachment:
mailselect.png
Description: PNG image
---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
