"C. Dominik Bódi": > (please CC me, I'm not subscribed to the list) > > Hello, > > I've discovered some strange behaviour on my Debian system, with > dm-crypt on raid 5 showing slow performance and I am seeking ways to > remedy that. > > My setup: > Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI) > Intel X58 chipset > Debian sid/unstable > kernels tried: 3.12.9, 3.13.4 (Debian kernels) > 4 x 3 TB SATA disks on a Intel X58 AHCI controller. > That controller has 6 SATA ports and connects directly to the chipsets > South Bridge, which connects to the North Bridge via a very fast DMI link. > Second PCIe-SATA controller on motherboard (ACHI driver, probably > JMicron chipset) - 2 SATA ports. > > On that disk I have a RAID5 array (source media are the 4th partition on > each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS > volume. That LUKS volume is a PV for LVM. > > 4 disks => RAID5 => LUKS Volume => LVM PV > > Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read > speed. Each single disk has about 170MB/s with hdparm, so that is ok. > > However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s. > Performing dd tests on the mounted LVM filesystems gets me about 160MB/s. *only* 150 ? there's lots of numbers here but maybe we should work backwards and KIS ask yourself what do you want to achieve ? when you know the results you want and/or need (2 different things) you can work backwards & figure out out how to get there do you *need* to have a certain read/write speed for your application or desired conclusion in mind ? do you have a recent version of cryptsetup with the benchmark option ? that can give you an idea of in memory performance and depends on your CPU & chosen ciphers/hashes i didn't quite get if your RAID is a hardware or software array ? > > Now I connected two of the four disks to the second SATA controller > > hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the > RAID5 array dropped to ca. 400MB/s. That was to be expected, as that > second controller seems somewhat slower. > > And now comes the funny part: > Now I did a hdparm -t /dev/mapper/encrypted_raid5 > and that got me ca. 300MB/s > > Running top whilst doing the hdparm tests showed me that, when all 4 > disks were on the Intel X58 AHCI controller, only 1 kworker thread was > running on a single CPU core, utilizing 100% CPU time. > > When 2 of those disks were connected to the second SATA controller, 2 > kworker threads were running on two different CPU cores. > > It seems that when the disks sit on two different controllers, dm-crypt > is using 2 kworker threads to handle the data, but uses only one kworker > thread when all disks sit on a single controller. i'll leave that to the experts ;-) > > My guess is that one single cpu core can decrypt/encrypt the LUKS volume > at about 150MB/s, which seems to be reasonable for that kind of CPU. > Thus, dm-crypt using more than one kworker threads leads to doubling of > the decryption speed, even if the two disks are slightly slower when > connected to the second controller. > > I'd like dm-crypt to use as many threads as possible (e.g. one kworker > for each disk in the RAID5 array). That would give me reasonable speeds > for the encrypted LUKS volume, close to that of the unencrypted RAID5 > volume. > > I've experimented with the RAID5 arrays group_thread_cnt sysfs setting. > That is set to zero by default, disabling multithreaded RAID5. However, > enabling that and setting it to 2 or 4 did not change anything speed-wise. > > I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity > setting, yet. Would that help? > > Is there a way to enforce dm-crypt using more than one kworker thread to > encrypt/decrypt data for a crypt-on-raid5 setup? > > Regards, > Dominik > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
