that's what i suspected, so i ran badblocks on all 6 hdds simultaneously, getting 100-110MB/s each(!). so the io subsystem is capable of ~600MB/s read, with very low iowait. when i start badblocks on all 6 crypto devices. the read speed drops to 29MB/s with high iowait. when i start it on just one crypto device, i got 100MB/s. start a second badblocks, the speed on both drops to 80MB/s. so the problem must be with dm_crypt or the way dm_crypt interacts with the underlying storage. maybe a problem/limitation when 6 crypto threads are running simultaneously? Am 09/14/2011 05:04 PM, schrieb Arno Wagner: > The RAID6 calculations should not matter much. For example > here is the RAID6 bootup-benchmark from my elderly > Athlon 64 X2 5600+: > ... > raid6: using algorithm sse2x2 (4867 MB/s) > > What I suspect for your problem on reading is an I/O > chip or attachment with asymmetrical performance. > This can happen, e.g. when the buffering is asymetrical > or one direction has a long switchover time when going > to/from different targets while the other does not. > Reads have to read one stripe in turn from each disk, > (with head movement, rotational latency, etc.) while > writes go to 1. OS buffer and 2. disk buffer. > > So writes are in principle faster on any Unix, as they > get buffered by the kernel and the disks, while reads do not. > Writes used to be slower because disk heads would need to > be positioned a lot better, but this difference has mostly > vanished in modern drives. > > Still, I suspect a hardware problem in the contoller(s) > that the disks attach to or their attachment to the system. > > Arno > > > On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: >> Hi, >> >> i've got a really strange problem with my setup, and i'm not quite sure >> why. >> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted >> with dm_crypt, on top of the encrypted block devices, there's a raid6. >> Normally you would assume, that writes are slower than reads, since >> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to >> calculate the 2 parity stripes). >> Let's take a look on the write performance first. Here's an average of >> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.12 0.05 94.15 2.17 0.00 3.50 >> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> avgrq-sz avgqu-sz await r_await w_await svctm %util >> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 >> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 >> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 >> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 >> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 >> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 >> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 >> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 >> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 >> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 >> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 >> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 >> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 >> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 >> >> Works as expected. Since this machine is powered by an quadcore, all >> this crypto and raid6 threads are nicely balanced over all 4 cores. >> Sequential Write averages out at about 300MB/s. >> So the CPU is capable of encrypting 6 times 72MB per second (while >> calculating parity for raid6). >> >> Now for the Reads. >> >> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 0.15 0.05 37.61 19.90 0.00 42.29 >> >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >> avgrq-sz avgqu-sz await r_await w_await svctm %util >> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 >> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 >> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 >> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 >> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 >> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 >> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 >> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 >> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 >> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 >> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 >> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 >> md3 0.00 0.00 40927.60 0.00 163710.40 0.00 >> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 >> >> 42% idle and almost 20% iowait? >> I've already tried everything i thought of. changed kernel CONFIG_HZ, >> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to >> ide mode, tuned read ahead for harddisks, crypto-devices and raid array >> (with different combinations). >> Same thing happens if i start badblocks on all 6 crypto devices. >> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD >> 910e) about 20%, the read rate increases at exactly 20%. >> Any advice would be greatly appreciated. >> >> thanks and best regards, Peter >> _______________________________________________ >> dm-crypt mailing list >> dm-crypt@xxxxxxxx >> http://www.saout.de/mailman/listinfo/dm-crypt > _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
