On Fri, 2003-11-07 at 14:49, XOR wrote:
> Greetings All!
> I have just put a PostFix+SpamAssassin system into production for a
> network of about 750,000 users running on Red Hat Enterprise Linux 3.0 ES.
> I have three front-end PostFix servers acting as mail relay's for all
> incoming SMTP traffic and four back-end SpamAssassin (spamd) boxes which the
> PostFix boxes are shipping mail too for scanning via a shell script I wrote.
> Everything is working wonderfully and mail is flowing quickly. I do have
> one concern, however, that I believe could become a problem in the near
> future. I wanted to get the list's thoughts on the solution I have come up
> with.
> The PostFix servers seem to be heavily loaded with disk I/O.
> Running a "top" shows that the processors spend the vast majority of their
> time in the "iowait" state - sometimes remaining at above 90% in this
> category for several minutes on end. Looking at a "vmstat 1" also shows
> that there is a hefty disk load, consistently reporting 1000+ in the "bo"
> field under the "io" heading. Running an "iostat -x 1" I see that write
> requests are waiting about 30ms to 40ms before being processed, sometimes
> jumping up close to 100ms during heavy usage (I can ping one of our routers
> at a remote site and get a response faster than I can write a byte of data
> to a local disk... heh). Disk service times are remaining somewhat low
> however, at consistently below 10ms, so once the data is written to disk
> it's at least being written quickly. I'm not sure how much I can trust the
> utilization percentage number, as I regularly see it jump to %150 to %160,
> but that's also a general indication that the disk subsystem is stressed.
> Turning off logging (and hence curtailing the truly massive amount of log
> data that was being written to disk) helped, but not quite enough.
> Unfortunately, the physical number of hard drives and their RAID
> configuration is suboptimal, and changing it isn't an option at this point.
> Fortunately, the machine has plenty of memory...
> ...which leads me to what I was considering. What about mounting
> the "/var/spool/postfix/incoming" queue and "/var/spool/postfix/active"
> queue as tmpfs with a max filesystem size of about 500MB a piece. I would
> mount the filesystem on boot in "/etc/fstab" (obviously) and then make the
> subdirectories needed in the queues ("A" through "0" - hex - they remain
> static as far as I can tell) then "chown" and "chmod" them to the
> appropriate owner and permissions in the PostFix startup script before
> PostFix actually starts. All file access in these two directories would be
> in physical memory, would take the load completely off of the disks, and be
> blazingly fast... as long as tmpfs doesn't try to use swap space in case of
> a physical memory shortage (which there most likely wouldn't be since
> there's a ton of memory in these boxes). Hell, even if there was a physical
> memory shortage and tmpfs had to use swap space, it would probably still be
> faster than reading and writing from/to the normal ext3 filesystem -
> according to a whitepaper over @ IBM that I read concerning tmpfs
> performance. The only downside of this setup I can think of is that in the
> very unlikely event of a server crash, all of the mail that was stored in
> the PostFix "incoming" and "active" queues would be lost. Poof. This,
> however, is an acceptable risk according to management.
> Thoughts?
>
> XOR
Seems like ther were performance issues with MailScnner and postfix that
were inherent in the way postfix handles the queues. Perhaps thei is
part of what you are seeing with SA alone too? Just a thought. As for
the rest sounds like you have it pretty well covered. Oh yeah get the
management approval in writing :)
Bret
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
