mhonarc 2.6.16: Out of Memory.

[Jim Barber <jim.barber@xxxxxxxxxxxxx>]

I've gone through the mail archives but can't find a solution to my problem.

I've been using mhonarc to process emails that are being generated by nightly builds of products within the company I work for.
This has been fine until recently when the developers enabled more verbose logging of the builds within their tool.
The mail messages that are being processed are stored in maildir format and they are sent as an email with a html attachment that is to be processed.
Looking at the raw format of the emails the html is encoded using base64.

The MS Visual Build Professional build tool is now producing ridiculously large HTML reports up to 34 MB in size (typical MS tool I guess).
Unfortunately I don't have control over that.
However the system I am running mhonarc on has 512MB of physical RAM, and 1024MB of swap allocated, so in theory should be able to process these files just fine.

The command I am using to process the folder is as follows:

	/usr/bin/mhonarc -add -quiet -savemem -rcfile /usr/local/etc/builds.mrc -outdir /var/www/build-reports /var/local/sharedfolders/.build/cur/ /var/local/sharedfolders/.build/new/

I was previously not using the -savemem parameter, but came across it on the mailing lists and gave it a shot but it still fails.
Unfortunately when mhonarc chews up all the memory, the kernel out-of-memory killer steps in and usually clobbers a mysql database running on the system that contains performance monitoring data.

The version of mhonarc I am using is 2.6.16 and my builds.mrc file referred to in the above command has the following contents (hopefully it doesn't wrap):

	<!-- MHonArc Resource File -->
	<!-- Run as:
	mhonarc -add -quiet -savemem -rcfile /usr/local/etc/builds.mrc -outdir /var/www/build-reports /var/local/sharedfolders/.build/cur/
	  -->

	<!-- Set Options. -->
	<Main>
	<NoThread>
	<Sort>
	<Reverse>
	<UseLocalTime>

	<!-- Since we will be pointing at Maildir directories, set the mhpattern option. -->
	<MHPATTERN>
	^[^\.]
	</MHPATTERN>

	<IdxFname>
	index.html
	</IdxFname>

	<!-- Expire messages after 30.5 days -->
	<EXPIREAGE>
	2635200
	</EXPIREAGE>

	<!-- Leave text/plain attachements as attachments and don't expand them inline. -->
	<MIMEArgs>
	m2h_text_plain::filter; attachcheck
	m2h_text_html::filter; allowcomments allowscript
	</MIMEArgs>

	<Title>
	Build Reports
	</Title>

	<!-- Format used for day group heading. -->
	<MsgLocalDateFmt>
	%B %d, %Y
	</MsgLocalDateFmt>

	<!-- Redefine LISTBEGIN since a table will be used for index listing. -->
	<ListBegin>
	<HR>
	<BR>
	<table border=1 frame="box" rules="groups" width=100%>
	</ListBegin>

	<!-- DAYBEGIN defines the markup to be printed when a new day group is started. -->
	<DayBegin>
	<tbody>
	<tr><td colspan=4><strong>$MSGLOCALDATE$</strong></td></tr>
	<tbody>
	</DayBegin>

	<!-- DAYEND defines the markup to be printed when a day group ends. No markup is needed in this case, so we leave it blank. -->
	<DayEnd>
	</DayEnd>

	<!-- Define LITEMPLATE to display the time of day the message was sent, message subject, author, and any annotation for the message.
	  -->
	<LiTemplate>
	<tr valign=top>
	<td>$MSGLOCALDATE(CUR;%H:%M)$</td>
	<td><b>$SUBJECT$</b></td>
	</tr>
	</LiTemplate>

	<!-- Define LISTEND to close table -->
	<ListEnd>
	</table>
	</ListEnd>

The current version of perl is 5.8.8.
The config above was set in conjunction with the developers to get it into a format that they are happy with.

The build folder has 1 weeks work of message which at the moment stands at 86 messages.
Most message have been up to around 2MB max in the past, but as of yesterday, it's now produced two massive 30MB messages, and a few smaller ones again.

When using top to monitor, mhonarc gets up to around 407MB or resident RAM size before crashing.
I can't remember the virtual memory size it allocated, but it was over 2GB.
These figures are pretty excessive for processing at most, a 34MB file.

Any solution on how to solve the problem would be very helpful.

Regards,

--
----------
Jim Barber
DDI Health