On Fri, Nov 19, 2010 at 05:48:22PM +0100, Michel Sébastien wrote: > > > Our biggest currently is about 30GB I think. > > >> I think the issue you will encounter first is clients will start to fall > >> down when folders exceed a 'reasonable' number of messages. Common IMAP > >> clients I've seen start to exhibit severe performance issues beyond a > >> few hundred thousand messages. > > > On a 32 bit architecture: we had one folder with over a million messages > > which was causing processes to run out of virtual memory trying to map > > the cache file in. This wouldn't be a problem with a 64 bit userland. > > very impressive to have so much messages in one folder therefor in one partition!. But with so many messages in one folder, I think that cyrus.index and even more cyrus.cache are huge. > Is mmap still efficient ? map a gigabit file should cost a lot of I/O and a relatively long reponse time to just access the records of the most recent emails. > > Is it time to break the design of one cyrus.index and cyrus.cache per folder by something more scalable ? To be honest - it doesn't actually hurt too badly once it's in memory cache. The cyrus.cache file isn't generally needed to be entirely read, and the secret of mmap is that you only read the bits you need as you need them - it's lazily loaded. The cyrus.index is still pretty small - about 100MB for a million messages. It doesn't take long to speed through that. Couple of seconds at most. There's no real answer if you're doing a sort on the messages, unless you go to multiple indexes (a la database engines). That's a whole different ballgame - but the the multiplier factor gets higher. For sane sizes of N (up to 20-30 thousand messages) the O(N) of the way Cyrus does it is cheaper than a more complex database. Bron. ---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
