On Tue, Apr 10, 2007 at 04:58:23PM +0200, Timo Schoeler wrote: > On Tue, 10 Apr 2007 06:56:43 -0500 > > 1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) > > 2. Which filesystem seems to be the better ? ext3 ? xfs ? > > reiserfs ? > > Do NEVER use XFS on GNU/Linux. (C)XFS is a brilliant FS on sgi's IRIX > machines, I never lost even a single in more than ten years. > > On GNU/Linux the implementation totally sucks. I'll stop my rant on > GNU/Linux now ;) > > My guess: ext3. ReiferFS has some very annoying weaknesses that may > affect you. We had the opposite experience. Turn off tails on Reiserfs and you'll still get better storage rates than ext3, and the difference in heavily loaded performance is amazing. We have machines that have been humming along just fine for months with significantly more users on them than they had with the abortive ext3 build. We also apply one patch to reiserfs. It's a one liner, using generic_write rather than the Hans Reiser special (5% faster, can deadlock under heavy load) We were in the process of working with Namesys people to get that one resolved back into the kernel when priorities got a little refocussed for them. Also, split meta is really valuable, with much faster for the meta partition. We have data on giant SATA RAID5 arrays (+ replication, + backups) and meta on 10kRPM SATA in RAID1. Here's 300 seconds worth of IO on one of our servers: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.05 11.17 5.35 5.95 55.94 136.92 27.97 68.46 17.07 0.00 0.42 0.22 0.24 sdb 0.01 128.52 43.72 50.74 687.77 1434.09 343.88 717.05 22.46 2.41 25.54 2.86 27.02 sdc 0.01 156.61 83.14 73.47 1219.23 1840.68 609.61 920.34 19.54 3.91 24.98 2.47 38.64 sdd 0.00 52.79 7.49 14.55 100.58 538.95 50.29 269.48 29.01 0.58 26.26 5.52 12.17 sde 4.55 52.16 17.98 16.58 218.33 549.94 109.17 274.97 22.24 1.36 39.47 5.11 17.67 sdf 0.33 255.53 99.87 36.73 3669.49 2334.67 1834.74 1167.34 43.95 1.62 11.85 2.76 37.64 sdg 0.00 60.41 31.85 31.18 552.26 732.64 276.13 366.32 20.38 1.63 25.92 4.40 27.76 sdh 2.28 51.95 23.36 11.57 1662.08 508.31 831.04 254.16 62.14 0.47 13.58 7.84 27.38 sdi 2.12 46.52 17.52 13.10 1457.10 476.69 728.55 238.34 63.14 0.45 14.68 6.44 19.71 sda is the system drive. The pairs are: meta: sdb sdc | sdf sdg data: sdd sde | sdh sdi Each 4 of which are an external drive unit with 4 fast small drives in RAID1 and 8 big slow drives in RAID5. Within the drives the layout is: [*-*-] [--*-] [*-v-v] [-v*v-] (bigger drives in the second unit) Where 'v' is a separate low-IO data storage partition that's not used for IMAP, '*' is a master and '-' is a replica. As you can imagine, the stats are a bit random there, but this is a pretty typical low-load time data set. It's only really daytime in Europe and parts of Asia at the moment, and we don't have as many users there as the US. > Best: ZFS on Solaris ;) Have the fixed the mmap problems yet? Otherwise, yeah - it looks pretty funky. I like the concepts. I'm also interested in reiser4 if/when it stabilises. It seems to have been designed with our workloads specifically in mind! There's also dualfs which was mentioned on the lkml recently that I'd love to play with if it's ever ported forward to the 2.6 series. > > 3. Which options to format the filesystem ? acording to the chosed > > filesystem No options. We mount: rw,noatime,nodiratime,notail,data=journal I'm not sure that notail is needed with more recent kernels, because there were patches that supposedly fixed the issue with that, but why mess with what works! > > 4. Which pop3 / imap proxy to use ? nginx. Without a doubt. Not only is it amazingly blindingly efficient with epoll (and probably kqueue if you went FreeBSD), but it has a very responsive and active author. Don't read the code though, it's very well written and tidy, but it will break your brain. Here's someone who ENJOYS writing state machines in C. > > 5. Single instance or multiple instances of cyrus ? taking in mind > > that there should be the option to recover a mailbox or some > > mail of a mailbox without having to shut down the whole cyrus system. I like small. It keeps the mailboxes.db small, and hence easier to scan in a hurry. If nothing else, it will improve IMAP LIST performance. That said, all users who share mailboxes will need to be in the same instance. > > 6. Best way to perform backups ? LVM snapshots ? shutting down some > > cyrus partitions ? RAID10 hot swap ? I'm in the middle of rewriting ours. It used to just be files, which was really easy because they never change(tm). It turns out not to be strictly true if someone deletes and recreates the folder, so the new one is going to be all UUID based (what with our MD5 UUID patch) and store cyrus.index files as well. We have a nice cyrus.index and cyrus.header parser, though I sure would be happier if mailboxes.db included the folder uniqueid in it. And cyrus.index for that matter. > > 7. Any other suggestion will be welcome. > > > > Thanks a lot !! Don't do it. It will make you pull your hair out. Email is full of spammers and scammers and it's a pointless waste of time. Tell your erstwhile users to find something better to do with their lives. Bron. ---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
