-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 (please CC me, I'm not subscribed to the list) Hello, I've discovered some strange behaviour on my Debian system, with dm-crypt on raid 5 showing slow performance and I am seeking ways to remedy that. My setup: Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI) Intel X58 chipset Debian sid/unstable kernels tried: 3.12.9, 3.13.4 (Debian kernels) 4 x 3 TB SATA disks on a Intel X58 AHCI controller. That controller has 6 SATA ports and connects directly to the chipsets South Bridge, which connects to the North Bridge via a very fast DMI link. Second PCIe-SATA controller on motherboard (ACHI driver, probably JMicron chipset) - 2 SATA ports. On that disk I have a RAID5 array (source media are the 4th partition on each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS volume. That LUKS volume is a PV for LVM. 4 disks => RAID5 => LUKS Volume => LVM PV Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read speed. Each single disk has about 170MB/s with hdparm, so that is ok. However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s. Performing dd tests on the mounted LVM filesystems gets me about 160MB/s. Now I connected two of the four disks to the second SATA controller hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the RAID5 array dropped to ca. 400MB/s. That was to be expected, as that second controller seems somewhat slower. And now comes the funny part: Now I did a hdparm -t /dev/mapper/encrypted_raid5 and that got me ca. 300MB/s Running top whilst doing the hdparm tests showed me that, when all 4 disks were on the Intel X58 AHCI controller, only 1 kworker thread was running on a single CPU core, utilizing 100% CPU time. When 2 of those disks were connected to the second SATA controller, 2 kworker threads were running on two different CPU cores. It seems that when the disks sit on two different controllers, dm-crypt is using 2 kworker threads to handle the data, but uses only one kworker thread when all disks sit on a single controller. My guess is that one single cpu core can decrypt/encrypt the LUKS volume at about 150MB/s, which seems to be reasonable for that kind of CPU. Thus, dm-crypt using more than one kworker threads leads to doubling of the decryption speed, even if the two disks are slightly slower when connected to the second controller. I'd like dm-crypt to use as many threads as possible (e.g. one kworker for each disk in the RAID5 array). That would give me reasonable speeds for the encrypted LUKS volume, close to that of the unencrypted RAID5 volume. I've experimented with the RAID5 arrays group_thread_cnt sysfs setting. That is set to zero by default, disabling multithreaded RAID5. However, enabling that and setting it to 2 or 4 did not change anything speed-wise. I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity setting, yet. Would that help? Is there a way to enforce dm-crypt using more than one kworker thread to encrypt/decrypt data for a crypt-on-raid5 setup? Regards, Dominik -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJTDND9AAoJEH57oErWeAO1IHgP/jghJGdRS/mWZ0SU3goeWSB3 0Bii9xRb0Eo1tPS+TTXRad308pCL8BXloNdgLJXL9T1P/TLOaX0e7QxVRCC3j6Uj siBZ7rMBp60bHQu6JNUyXPwssgQhyRhL7Web2tHm7RQYDUoNaVdIjL0p4KvCRAX6 4RE2FzMEze/DgumZtXYgSlBBxVb9Za87Y/iwH+SJQ87kloKPkD7kgX2hByYJ5D7h NswCns1slj5KYyOSqikcoJM1Z9Sd1eJjMhRJ37/hH5ERMiP7m+LMO07ZsxtPCdE0 Fyi1J6Ra28YzKJPTcaBSa2oy+zeauennCIgLbA2px+7t3zEoaUiMSg4WrFbrV8i4 x1WWiHHrZV23dza2Hgy4BcE1lXAGxEKCuOSZr0Js/b20DfCeoV84oc4GIlMo3ZL4 LztRJ4yrtOUehH9KUwTqFLFk3r9xWDMC3ZlrbyyoxVAB5Nr+w1FnAzQRnvC7DOHb wU2lj9zWloNT8hUsCwtdpLxQWSj9iWw1j70xuqSUGIP3SLBgBQdUfpZP0Xwwe0Bi 577yt6CbQcaqGgjhtlgetwL9e8Vj8rMXbKZUKbyimimwmHEAJ++pdOWnBTdz7NOQ IKXP29mbI7+uIrzcgQvH6o5NS6psxdIcuxjfgYoR8CLPcNB87xKDcMXF9hU8JkZA fVE8Z2X5LPDksJrs3eb4 =5Mww -----END PGP SIGNATURE----- _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
