|
Hi, i've got a really strange problem with my setup, and i'm not quite sure why. First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted with dm_crypt, on top of the encrypted block devices, there's a raid6. Normally you would assume, that writes are slower than reads, since dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to calculate the 2 parity stripes). Let's take a look on the write performance first. Here's an average of 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M" avg-cpu: %user %nice %system %iowait %steal %idle 0.12 0.05 94.15 2.17 0.00 3.50 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 9.30 15200.00 1.80 2972.70 44.40 73472.40 49.43 15.53 5.21 201.72 5.09 0.26 76.68 sdd 6.30 14495.00 1.60 3759.60 31.60 73756.00 39.24 2.91 0.77 133.44 0.71 0.19 73.08 sdf 3.10 14519.30 1.70 3650.00 19.20 73429.20 40.23 2.86 0.78 137.06 0.72 0.20 72.52 sde 6.10 14240.10 1.30 3964.20 29.60 73572.40 37.12 2.80 0.70 173.38 0.65 0.18 71.70 sdg 3.10 14859.40 1.10 3306.50 16.80 73410.00 44.40 2.72 0.82 3.45 0.82 0.19 63.06 sdc 0.20 14276.20 1.00 3909.40 4.80 73450.00 37.57 2.65 0.67 41.80 0.66 0.17 65.29 md3 0.00 0.00 0.00 4609.60 0.00 294870.40 127.94 0.00 0.00 0.00 0.00 0.00 0.00 Works as expected. Since this machine is powered by an quadcore, all this crypto and raid6 threads are nicely balanced over all 4 cores. Sequential Write averages out at about 300MB/s. So the CPU is capable of encrypting 6 times 72MB per second (while calculating parity for raid6). Now for the Reads. 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M" avg-cpu: %user %nice %system %iowait %steal %idle 0.15 0.05 37.61 19.90 0.00 42.29 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 5700.20 0.00 1141.90 0.00 27507.20 0.00 48.18 6.37 5.52 5.52 0.00 0.42 48.27 sdd 5420.60 0.00 1419.80 0.00 27462.40 0.00 38.68 4.47 3.14 3.14 0.00 0.30 42.12 sdf 5646.90 0.00 1198.10 0.00 27418.00 0.00 45.77 5.50 4.57 4.57 0.00 0.36 42.91 sde 5837.10 0.00 1006.40 0.00 27507.20 0.00 54.66 6.69 6.61 6.61 0.00 0.49 49.45 sdg 5671.10 0.00 1171.30 0.00 27401.20 0.00 46.79 4.65 3.95 3.95 0.00 0.34 39.99 sdc 5994.20 0.00 851.30 0.00 27485.60 0.00 64.57 7.11 8.27 8.27 0.00 0.57 48.44 md3 0.00 0.00 40927.60 0.00 163710.40 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 42% idle and almost 20% iowait? I've already tried everything i thought of. changed kernel CONFIG_HZ, tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to ide mode, tuned read ahead for harddisks, crypto-devices and raid array (with different combinations). Same thing happens if i start badblocks on all 6 crypto devices. Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD 910e) about 20%, the read rate increases at exactly 20%. Any advice would be greatly appreciated. thanks and best regards, Peter |
_______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
