Re: Creating a Mhonarc archive from MS Outlook emails

[Rick Boatright <boatright@xxxxxxxxxxxxxx>]

I have 72,000 messages in a single mhonarc archive running on a dual 
xeon 800 MHz windows 2000 machine with 500 Meg of Ram.  The result files 
include nearly 800 each thread, author and date index pages, along with 
the 72000 msg*.html files all in a single directory.

The last complete generate (back in Sept of '03) took 4 days.  The 
_next_ version will be using the mharc project technqiues to chop it 
into bits.  

Interestingly IIS, and Windows 2000 handle the large directory just 
fine, and responsiveness is not a problem.  The system is PRIMARILY 
accessed by a Namazu search, which ALSO works fine, with sub-second 
response to any query.

Rick Boatright

Earl Hood wrote:

> On March 15, 2004 at 16:49, "Lyle Schofield" wrote:
>
>  
>
> > Side question -- are there any practical limits to the number of
> > messages MHonArc can convert into an archive?  I've been testing in
> > batches of 350, and it only takes a few minutes on my laptop.  But, if I
> > try to dump thousands of messages in there will I hit some limit of what
> > can be processed?  Thanks in advance.
> >    
>
> Scaling can be a problem if your resources are limitied.  When MHonArc
> processes an archive, all the meta-information of the archive
> is loaded, so archive updates for large archives will slow down
> over time.  Therefore, doing real-time updates (i.e. add messages
> right as they are received) of large archives should not be done to
> avoid bottleneck problems.
>
> The general solution for dealing with a large archive is to
> break it down into smaller archives denoted by fixed time
> periods (like each week, month, or year).  The mharc project,
> <http://www.mhonarc.org/mharc/> provides such capabilites, and other
> users have developed their own (search the list archives).
>
> If you are archives will grow to an indefinite size, then I highly
> recommend going with the smaller sub-archive approach.
>
> --ewh
>
> P.S.  I have heard of users that have had up to 60,000 messages in a
> single archive.  But I think you need some decent hardware and RAM
> to effectively work with such a beast.
>
>
>