Hi All,
I've just spent the last few hours reviewing all the archived list traffic
for 2006, looking for pearls of wisdom on using loop-aes, but my current
problem seems to be little known, if at all. I've been using loop-aes on
my file servers for a couple of years now, and have recently shifted to
using AMD AM2 processors because of their modest cost & power, along with
the nice feature of directly supporting ECC DDR2 memory.
On my penultimate file server, I used 5 Seagate 250G Sata drives with an AMD
3800+ AM2 CPU, and this got fairly (I thought) decent performance. I did a
fair amount of benchmarking, and got 300 MB/s writes and 200MB/s reads with
a 5 disk raid0 and XFS. Moving to RAID6 and Reiser3.6 dropped the
performance to 71 MB/s read and 69 MB/s write. (All benchmarks with
Bonnie++ 1.03 using 4G data chunks).
Adding loop-AES-v3.1b with AES256 as a layer between Reiser3.6 and RAID6
(with Dapper Drake Ubuntu server, kernel 2.6.16-26, AMD64) dropped the
performance down to 32MB/s write and 34 MB/s read. Not great, but
tolerable, since the machine was only connected to a 100 Mbit ethernet
router.
Now I've built a totally new machine using an ASUS M2N-E motherboard with
the NForce 570 Ultra chip set. It has 2 GBytes of PC5300 ECC DDR2 memory,
and six Seagate 400 GB SATA2 drives. The CPU is a AMD 4200+ X2 dual core
AM2 with dual 512K L2 caches.
I installed Edgy AMD64 (2.6.17-10-generic #2 SMP ) Kubuntu desktop on a 7th
drive, and have since been beating the raid array senseless with benchmarks,
trying to get a decent throughput. Before installing the SATA drives, I
ran some benchmarks on the 7th drive, an older 160GB WD PATA drive.
I got 57 MB/s write and 58 MB/s read performance on a PATA drive partition
with Reiser3.6 and loop-AES-v3.1d. This seems OK. Not great, but OK.
But it all went to hell when I tried the full stack on the six drive RAID5
or RAID6 array. Performance for writing fell to a pitiful 15 Mbytes/sec.
I theorized that maybe the RAID helper thread was thrashing the caches, so I
reduced the raid array chunksize to 16K, then 8K, then 4K. This didn't help
at all. I increased the drive readahead to 1024 sectors, which seemed to
help the read performance, but did nothing for write performance. The SATA2
drives are already using 16 sector transfers, which appears to be the
maximum possible. I've tried ext3, xfs, jfs and reiser3.6 filesystems,
nothing seems to help. The best write speed I've gotten so far is 17
Mbyte/s with RAID6 and 64K chunksizes on a reiser3.6 filesystem.
RAID5 performance is terrible when I stack loop-AES on top of it. Without
loop-AES, I'm getting 99 Mbytes/sec Read, and 105 Mbytes Write with RAID5.
This is the performance level I want to hit, since it's just about right
when transferring data with Gigabit ethernet.
My next step is to install Ubuntu AMD64 server on a spare partition and see
if getting rid of X and KDE will help at all. I've got about 121 tasks
running with Kubuntu desktop, and only 64 with the Ubuntu server on the
other machine. I suspect my cache runneth low, but I'm not sure how to
prove this is the problem, other than by switching the OS. Looking at the
.config file in the kernel, it looks like deadline IO scheduling is the
default.
Anybody got any ideas on how to find the problem? I suspect that if I shell
out $500 for an ARC-1220 raid controller, I can get the performance back up
to 50 or 60 Mbytes/sec, but this seems to defeat the purpose of building a
low cost Linux file server. I'm beginning to wonder if I should try another
distro. Knoppix 5.1.1 will be an easy test, certainly.
Best regards to all, and many thanks to Jari Ruusu for all his fine work.
George Koss
_________________________________________________________________
Type your favorite song. Get a customized station. Try MSN Radio powered
by Pandora. http://radio.msn.com/?icid=T002MSN03A07001
-
Linux-crypto: cryptography in and on the Linux system
Archive: http://mail.nl.linux.org/linux-crypto/
