-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, this query is best suited for md-raid mailing list as what you measure: dd if=/dev/md3 of=/dev/null bs=1M does not involve any decryption. You are reading directly from the md device and there is no matter what is actually there. Cheers, - -Nik On 09/14/2011 07:47 PM, Peter Merhaut wrote: > that's what i suspected, so i ran badblocks on all 6 hdds > simultaneously, getting 100-110MB/s each(!). > so the io subsystem is capable of ~600MB/s read, with very low iowait. > when i start badblocks on all 6 crypto devices. the read speed drops to > 29MB/s with high iowait. > > when i start it on just one crypto device, i got 100MB/s. start a second > badblocks, the speed on both drops to 80MB/s. > > so the problem must be with dm_crypt or the way dm_crypt interacts with > the underlying storage. maybe a problem/limitation when 6 crypto threads > are running simultaneously? > > Am 09/14/2011 05:04 PM, schrieb Arno Wagner: >> The RAID6 calculations should not matter much. For example >> here is the RAID6 bootup-benchmark from my elderly >> Athlon 64 X2 5600+: >> ... >> raid6: using algorithm sse2x2 (4867 MB/s) >> >> What I suspect for your problem on reading is an I/O >> chip or attachment with asymmetrical performance. >> This can happen, e.g. when the buffering is asymetrical >> or one direction has a long switchover time when going >> to/from different targets while the other does not. >> Reads have to read one stripe in turn from each disk, >> (with head movement, rotational latency, etc.) while >> writes go to 1. OS buffer and 2. disk buffer. >> >> So writes are in principle faster on any Unix, as they >> get buffered by the kernel and the disks, while reads do not. >> Writes used to be slower because disk heads would need to >> be positioned a lot better, but this difference has mostly >> vanished in modern drives. >> >> Still, I suspect a hardware problem in the contoller(s) >> that the disks attach to or their attachment to the system. >> >> Arno >> >> >> On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote: >>> Hi, >>> >>> i've got a really strange problem with my setup, and i'm not quite sure >>> why. >>> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted >>> with dm_crypt, on top of the encrypted block devices, there's a raid6. >>> Normally you would assume, that writes are slower than reads, since >>> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to >>> calculate the 2 parity stripes). >>> Let's take a look on the write performance first. Here's an average of >>> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 0.12 0.05 94.15 2.17 0.00 3.50 >>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >>> sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 >>> 49.43 15.53 5.21 201.72 5.09 0.26 76.68 >>> sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 >>> 39.24 2.91 0.77 133.44 0.71 0.19 73.08 >>> sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 >>> 40.23 2.86 0.78 137.06 0.72 0.20 72.52 >>> sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 >>> 37.12 2.80 0.70 173.38 0.65 0.18 71.70 >>> sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 >>> 44.40 2.72 0.82 3.45 0.82 0.19 63.06 >>> sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 >>> 37.57 2.65 0.67 41.80 0.66 0.17 65.29 >>> md3 0.00 0.00 0.00 4609.60 0.00 294870.40 >>> 127.94 0.00 0.00 0.00 0.00 0.00 0.00 >>> >>> Works as expected. Since this machine is powered by an quadcore, all >>> this crypto and raid6 threads are nicely balanced over all 4 cores. >>> Sequential Write averages out at about 300MB/s. >>> So the CPU is capable of encrypting 6 times 72MB per second (while >>> calculating parity for raid6). >>> >>> Now for the Reads. >>> >>> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 0.15 0.05 37.61 19.90 0.00 42.29 >>> >>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s >>> avgrq-sz avgqu-sz await r_await w_await svctm %util >>> sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 >>> 48.18 6.37 5.52 5.52 0.00 0.42 48.27 >>> sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 >>> 38.68 4.47 3.14 3.14 0.00 0.30 42.12 >>> sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 >>> 45.77 5.50 4.57 4.57 0.00 0.36 42.91 >>> sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 >>> 54.66 6.69 6.61 6.61 0.00 0.49 49.45 >>> sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 >>> 46.79 4.65 3.95 3.95 0.00 0.34 39.99 >>> sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 >>> 64.57 7.11 8.27 8.27 0.00 0.57 48.44 >>> md3 0.00 0.00 40927.60 0.00 163710.40 0.00 >>> 8.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> >>> 42% idle and almost 20% iowait? >>> I've already tried everything i thought of. changed kernel CONFIG_HZ, >>> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to >>> ide mode, tuned read ahead for harddisks, crypto-devices and raid array >>> (with different combinations). >>> Same thing happens if i start badblocks on all 6 crypto devices. >>> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD >>> 910e) about 20%, the read rate increases at exactly 20%. >>> Any advice would be greatly appreciated. >>> >>> thanks and best regards, Peter >>> _______________________________________________ >>> dm-crypt mailing list >>> dm-crypt@xxxxxxxx >>> http://www.saout.de/mailman/listinfo/dm-crypt >> > > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJOccpBAAoJEDFLYVOGGjgXOMAH/24/+1Q1hIbo0bPfIyDdGums SREqEsEz+ty/eOpaoKyjnFM0Ec0RF6ktQ5eVCQndje0SZp0R7ViQueooa8Y5GPBc 4brlPx7hJJ2MD9qsv7u5QOhT5BzClFaD891efN5O11VYuz25aioVWw6aMJlXfu4Y Mc4lVfgpyUwMXnLijtZYXxucGcJlrdGeEq7TcZ38DmM+hhhWzYfAWIlC5YNW7Jvs TvNUbIRe6KjHvYwyaG82n+9OgqGkcv303iNLEzqTypyVu7oBtPU1C9B7+3QEwwXn 857kyuxXmq2f1BtXJzleXknng2x3YgyqT0wmN3CK1jBVMmz87ONrkryFLBUtgCU= =jjZC -----END PGP SIGNATURE----- _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
