Hi how does that solve my issue with e.g. the extremely slow badblocks performance on the crypto-devices? (absolutely NO md involved!) best regards, Peter Am 09/15/2011 11:49 AM, schrieb Nikolay Kichukov: > Hi, > this query is best suited for md-raid mailing list as what you measure: > > dd if=/dev/md3 of=/dev/null bs=1M > > does not involve any decryption. You are reading directly from the md > device and there is no matter what is actually there. > > Cheers, > -Nik > > On 09/14/2011 07:47 PM, Peter Merhaut wrote: > > that's what i suspected, so i ran badblocks on all 6 hdds > > simultaneously, getting 100-110MB/s each(!). > > so the io subsystem is capable of ~600MB/s read, with very low iowait. > > when i start badblocks on all 6 crypto devices. the read speed drops to > > 29MB/s with high iowait. > > > when i start it on just one crypto device, i got 100MB/s. start a second > > badblocks, the speed on both drops to 80MB/s. > > > so the problem must be with dm_crypt or the way dm_crypt interacts with > > the underlying storage. maybe a problem/limitation when 6 crypto threads > > are running simultaneously? > > > Am 09/14/2011 05:04 PM, schrieb Arno Wagner: > >> The RAID6 calculations should not matter much. For example > >> here is the RAID6 bootup-benchmark from my elderly > >> Athlon 64 X2 5600+: > >> ... > >> raid6: using algorithm sse2x2 (4867 MB/s) > >> > >> What I suspect for your problem on reading is an I/O > >> chip or attachment with asymmetrical performance. > >> This can happen, e.g. when the buffering is asymetrical > >> or one direction has a long switchover time when going > >> to/from different targets while the other does not. > >> Reads have to read one stripe in turn from each disk, > >> (with head movement, rotational latency, etc.) while > >> writes go to 1. OS buffer and 2. disk buffer. > >> > >> So writes are in principle faster on any Unix, as they > >> get buffered by the kernel and the disks, while reads do not. > >> Writes used to be slower because disk heads would need to > >> be positioned a lot better, but this difference has mostly > >> vanished in modern drives. > >> > >> Still, I suspect a hardware problem in the contoller(s) > >> that the disks attach to or their attachment to the system. > >> > >> Arno > >> > >> > >> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: > >>> Hi, > >>> > >>> i've got a really strange problem with my setup, and i'm not quite > sure > >>> why. > >>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted > >>> with dm_crypt, on top of the encrypted block devices, there's a raid6. > >>> Normally you would assume, that writes are slower than reads, since > >>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 > has to > >>> calculate the 2 parity stripes). > >>> Let's take a look on the write performance first. Here's an average of > >>> 10 seconds iostat while running "dd if=/dev/zero > of=/daten/testfile bs=1M" > >>> > >>> avg-cpu: %user %nice %system %iowait %steal %idle > >>> 0.12 0.05 94.15 2.17 0.00 3.50 > >>> > >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > >>> avgrq-sz avgqu-sz await r_await w_await svctm %util > >>> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 > >>> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 > >>> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 > >>> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 > >>> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 > >>> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 > >>> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 > >>> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 > >>> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 > >>> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 > >>> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 > >>> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 > >>> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 > >>> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 > >>> > >>> Works as expected. Since this machine is powered by an quadcore, all > >>> this crypto and raid6 threads are nicely balanced over all 4 cores. > >>> Sequential Write averages out at about 300MB/s. > >>> So the CPU is capable of encrypting 6 times 72MB per second (while > >>> calculating parity for raid6). > >>> > >>> Now for the Reads. > >>> > >>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" > >>> > >>> avg-cpu: %user %nice %system %iowait %steal %idle > >>> 0.15 0.05 37.61 19.90 0.00 42.29 > >>> > >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > >>> avgrq-sz avgqu-sz await r_await w_await svctm %util > >>> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 > >>> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 > >>> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 > >>> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 > >>> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 > >>> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 > >>> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 > >>> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 > >>> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 > >>> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 > >>> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 > >>> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 > >>> md3 0.00 0.00 40927.60 0.00 163710.40 > 0.00 > >>> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>> > >>> 42% idle and almost 20% iowait? > >>> I've already tried everything i thought of. changed kernel CONFIG_HZ, > >>> tried a tickless Kernel, enabled/disabled ncq, changed bios from > ahci to > >>> ide mode, tuned read ahead for harddisks, crypto-devices and raid > array > >>> (with different combinations). > >>> Same thing happens if i start badblocks on all 6 crypto devices. > >>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD > >>> 910e) about 20%, the read rate increases at exactly 20%. > >>> Any advice would be greatly appreciated. > >>> > >>> thanks and best regards, Peter > >>> _______________________________________________ > >>> dm-crypt mailing list > >>> dm-crypt@xxxxxxxx > >>> http://www.saout.de/mailman/listinfo/dm-crypt > >> > > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@xxxxxxxx > > http://www.saout.de/mailman/listinfo/dm-crypt _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
