Hello again, > Von: mbroz@xxxxxxxxxx [mailto:mbroz@xxxxxxxxxx] > I do not want to speculate what exactly causes latency issues > - it is not one thing (testing of per-cpu worker threads > changes shows that page cache and fs are also important part > which can contribute to some problems here etc.) I reran the test on 2.6.32 again and looked at vmstat for the test run and found something interesing: Run1: ext3 with /dev/sdX backend Run2: ext3 with dmcrypt backend (dm_crypt is backed by the same /dev/sdX) Run3: xfs with dmcrypt backend (dm_crypt is backed by the same /dev/sdX) Run2 has a very stuttery I/O pattern, no constant I/O flow. For me this could explain the result of -35% performance für ext3 on dm_crypt. While Run1) and Run3) work in a pipeline fashion (where latency is not so much of an issue) - Run2 clearly distinguished different phases (write to buffer cache / encrypt / write to disk). Here tiny latency increases in the I/O will pile up in phase (2) encrypt. This looks like a very odd behaviour / incompatiblity to me :/ Unfortunately I'm not a kernel expert, so I can't debug further. Regards, Robert ---- Run1: ====== ext3: Run on un-encrypted backend (plain disk): procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 3621816 131712 3504132 0 0 8 2790 196 345 2 5 82 10 1 2 0 0 3521500 135872 3586616 0 0 24 4 712 151 2 26 65 0 7 0 3 0 3467760 138568 3628056 0 0 16 54916 2792 2003 2 17 11 69 1 0 3 0 3408284 142220 3673428 0 0 12 46132 3516 2526 2 19 1 77 1 1 2 0 3343768 145108 3726176 0 0 12 47828 2965 2206 2 15 1 78 4 0 3 0 3271328 148116 3786512 0 0 12 45212 3087 2218 2 17 1 77 4 1 2 0 3187632 151352 3856728 0 0 12 45992 3215 2370 2 17 0 80 1 0 3 0 3095036 154432 3936248 0 0 16 45888 3215 2150 2 17 10 72 0 2 2 0 2988832 157912 4026676 0 0 8 39804 2734 1066 2 22 47 30 0 1 1 0 2937600 160608 4066824 0 0 16 36164 2524 1532 1 15 12 69 2 1 1 0 2877724 164160 4115332 0 0 40 48636 1476 208 3 21 23 50 4 1 1 0 2789056 168604 4187808 0 0 8 45516 1923 238 2 23 13 62 0 1 2 0 2722748 171612 4244048 0 0 16 48520 1855 268 1 17 29 50 2 0 3 0 2647884 174764 4307504 0 0 12 45768 2446 2401 2 18 23 54 3 2 2 0 2563828 177716 4380532 0 0 12 37536 2713 2080 2 16 0 80 1 1 3 0 2463276 181088 4465828 0 0 12 44952 3320 2230 3 20 0 75 2 1 2 0 2385192 184224 4530712 0 0 12 47352 2708 2307 1 18 0 81 0 0 3 0 2323952 187452 4577956 0 0 12 47512 3346 2289 3 18 0 77 2 2 2 0 2268744 190812 4620344 0 0 12 47636 3346 2600 2 17 0 81 0 1 2 0 2177296 194916 4695412 0 0 8 49964 3625 2421 2 21 0 74 3 1 2 0 2648832 195028 4265308 0 0 0 29352 1665 1643 2 32 0 64 2 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 3 0 3068280 195028 3865160 0 0 0 34360 1922 2481 1 29 9 61 0 0 3 0 3103564 195076 3834508 0 0 16 48092 1925 1621 0 4 42 54 0 0 3 0 3139772 195116 3801664 0 0 12 47444 2049 1845 0 3 52 44 0 0 3 0 3196236 195120 3764352 0 0 0 39396 1852 2226 0 4 40 55 0 0 3 0 3444792 195200 3531164 0 0 16 38064 2391 1594 1 11 55 33 0 1 1 0 3736860 195224 3269744 0 0 4 23356 2991 700 1 19 54 25 2 1 1 0 4244964 195224 2803476 0 0 0 18492 3625 435 1 33 4 62 0 0 1 0 4254208 195328 2796780 0 0 20 50000 1431 112 0 3 46 51 0 1 0 0 4532788 97148 2725008 0 0 1 136 835 458 17 6 66 9 2 1 0 0 4532712 97148 2724992 0 0 41 5 807 93 32 0 65 0 2 Run2 ====== ext3: Same run with encrypted Backend (dm_crypt, aes_cbc) 2 1 0 4161032 412308 2725000 0 0 40 315112 1463 1648 2 23 66 2 6 2 2 0 4166480 412308 2725004 0 0 0 0 2844 2727 0 23 50 25 1 1 1 0 4174364 412308 2725004 0 0 0 0 4138 3636 0 26 41 30 3 1 1 0 4183044 412308 2725004 0 0 0 0 1887 1932 0 24 60 14 2 1 1 0 4192220 412308 2725004 0 0 0 0 2335 2264 0 29 61 10 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 1 0 4200280 412316 2724996 0 0 0 28 2063 2155 0 23 59 18 0 1 1 0 4533280 97208 2725084 0 0 105 4 1018 1241 0 8 54 39 0 2 0 0 4438560 98168 2809272 0 0 32 108 844 5537 16 14 65 2 2 1 0 0 4400668 100340 2838468 0 0 16 0 718 10816 23 9 65 1 2 1 0 0 4367456 102364 2864068 0 0 8 0 680 8588 22 7 66 0 4 1 0 0 4336292 104296 2887836 0 0 16 0 690 8833 19 7 67 1 5 2 0 0 4303192 106016 2913276 0 0 8 0 685 9194 21 9 66 0 4 1 2 0 4271408 106300 2916456 0 0 8 191356 1580 961 3 26 37 31 3 1 2 0 4279096 106300 2916464 0 0 0 0 1775 87 0 17 59 24 0 1 2 0 4286164 106300 2916464 0 0 0 0 1947 90 0 18 60 22 0 0 3 0 4291620 106300 2916464 0 0 0 0 1704 91 0 15 58 27 0 1 0 0 4283820 107044 2930500 0 0 16 9064 977 6038 14 10 48 26 2 1 0 0 4266336 107648 2945788 0 0 0 17576 1116 7134 18 10 47 23 2 1 0 0 4235284 108668 2972812 0 0 8 0 728 12201 26 6 64 1 3 1 0 0 4204896 109732 2999168 0 0 8 0 743 11814 27 6 63 0 3 1 0 0 4176104 110716 3024536 0 0 0 0 762 11966 27 6 66 0 2 1 0 0 4147956 111676 3049548 0 0 8 56 791 11842 26 6 67 0 1 2 1 0 4106776 112384 3068496 0 0 0 119492 1386 8563 26 28 10 22 14 1 2 0 4113688 112408 3069244 0 0 4 0 1936 471 1 20 0 78 1 1 0 0 4121364 112536 3071740 0 0 8 4688 1689 1174 2 16 5 72 4 1 0 0 4097144 113432 3093496 0 0 0 4092 1151 10074 23 9 54 9 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 4070032 114444 3117072 0 0 8 0 709 10846 23 6 65 1 6 2 1 0 4038420 115340 3136936 0 0 0 66320 1681 9014 30 39 7 11 13 0 3 0 4029340 115756 3147208 0 0 4 29604 1838 5212 14 30 2 39 15 1 0 0 4018948 116400 3161668 0 0 8 2772 2072 6793 14 8 29 48 1 1 0 0 3989892 117436 3187080 0 0 0 0 763 11976 27 6 66 0 1 1 0 0 3959948 118748 3212344 0 0 8 0 745 10322 24 7 66 1 3 3 2 0 3930692 119544 3228896 0 0 0 75952 1560 7265 25 34 7 19 15 1 1 0 3914316 120640 3249400 0 0 12 4496 1569 9340 27 23 25 17 8 1 0 0 3872148 122724 3290044 0 0 8 4 749 10154 21 11 66 1 2 1 0 0 3754576 127996 3386976 0 0 24 0 748 212 5 28 66 1 0 1 0 0 3662420 133292 3457428 0 0 32 0 770 285 6 26 66 2 0 1 2 0 3607292 134544 3471488 0 0 4 247188 1545 114 1 30 4 59 7 1 2 0 3613864 134544 3471488 0 0 0 0 1899 93 0 17 0 83 0 1 2 0 3620684 134552 3471480 0 0 0 16 1810 96 0 17 0 83 0 1 2 0 3626636 134552 3471488 0 0 0 0 1543 83 0 13 0 87 0 1 2 0 3633580 134552 3471488 0 0 0 0 1835 88 0 16 0 84 0 1 1 0 3628936 135296 3478356 0 0 24 19200 1659 115 1 24 9 63 4 1 1 0 3554984 138016 3538396 0 0 8 77976 1143 116 2 43 13 25 16 0 2 0 3554612 138016 3538428 0 0 0 0 1029 54 0 9 54 37 0 0 2 0 3552644 138404 3545736 0 0 0 0 1397 95 1 6 50 43 0 0 2 0 3545124 138752 3554636 0 0 4 0 2847 171 0 8 50 41 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 3418976 143204 3665872 0 0 20 0 884 126 3 24 63 9 0 1 2 0 3344964 144892 3709656 0 0 4 167920 1498 110 0 38 10 45 7 1 2 0 3352404 144892 3709656 0 0 0 0 2006 99 0 20 4 75 0 1 2 0 3360216 144892 3709656 0 0 0 0 1963 95 0 19 4 77 0 1 0 0 3361600 145276 3716428 0 0 16 11456 1368 114 0 16 22 61 2 1 0 0 3232156 150392 3827480 0 0 16 0 737 186 5 27 67 1 1 1 2 0 3165872 152620 3863048 0 0 12 160908 1496 141 3 36 38 14 8 0 3 0 3173188 152620 3863048 0 0 0 0 1929 93 0 20 60 20 0 1 2 0 3180628 152620 3863048 0 0 0 0 1980 98 0 19 61 20 0 1 2 0 3152568 154376 3884704 0 0 16 40216 1490 106 1 37 34 18 10 1 0 0 3139400 155832 3901336 0 0 4 20 2173 167 1 11 18 70 0 1 2 0 3095536 157908 3932828 0 0 12 47652 1582 148 3 35 9 45 9 1 0 0 3005852 162460 4012048 0 0 24 5452 852 188 3 26 56 12 3 1 0 0 2853664 167780 4143148 0 0 16 0 777 176 5 28 66 1 0 2 0 0 2715040 172524 4259044 0 0 16 5704 809 197 3 31 62 1 3 2 1 0 2613312 175684 4317960 0 0 12 243488 1553 113 3 56 16 10 16 0 2 0 2592604 175684 4317944 0 0 0 158036 1681 82 0 20 12 65 3 1 1 0 2600292 175684 4317944 0 0 0 0 2078 94 0 20 5 75 0 1 1 0 2606244 175684 4317944 0 0 0 0 1622 79 0 16 3 81 0 0 2 0 2613560 175700 4317932 0 0 4 16 1940 108 0 19 4 77 0 1 1 0 2620752 175700 4317944 0 0 0 0 1892 90 0 18 4 78 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 2 0 2626332 175700 4317944 0 0 0 0 1582 74 0 15 4 81 0 0 2 0 2632284 175700 4317944 0 0 0 0 1612 81 0 15 4 81 0 0 2 0 2642452 175700 4317944 0 0 0 0 1708 82 0 13 3 83 0 2 1 0 2626680 176584 4331580 0 0 0 17220 2422 140 1 19 5 69 6 1 0 0 2589492 179104 4363236 0 0 28 19496 975 143 2 27 17 39 14 1 0 0 2495704 184692 4436920 0 0 24 0 753 284 6 27 66 1 0 1 0 0 2351244 190080 4562020 0 0 16 0 761 189 4 29 66 1 0 Run3: ======= xfs: Now a xfs on the same device (dm_crypt, aes_cbc) 0 0 0 4532784 97404 2724988 0 0 0 0 32 12 0 0 100 0 0 0 0 0 4532816 97404 2724992 0 0 0 0 31 11 0 0 100 0 0 0 0 0 4532816 97404 2724992 0 0 0 0 28 17 0 0 100 0 0 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 4532792 97412 2724984 0 0 0 68 211 340 1 1 97 0 0 0 1 0 4533312 97428 2724980 0 0 20 10574 945 849 1 7 55 36 1 1 0 0 4446004 97420 2805168 0 0 179 4404 1119 3817 11 9 73 5 2 1 0 0 4423212 97420 2823960 0 0 0 6984 1101 7801 19 10 60 0 11 1 0 0 4392896 97420 2847932 0 0 0 10924 1408 8646 20 15 61 0 4 1 0 0 4370208 97420 2867048 0 0 0 7644 997 6719 15 9 73 0 3 1 0 0 4343520 97420 2888740 0 0 0 10768 1236 7424 21 13 58 0 8 1 0 0 4317964 97428 2909456 0 0 0 11408 1335 6238 20 12 59 0 9 1 0 0 4297124 97428 2926644 0 0 0 10040 1158 5890 16 10 74 0 1 1 0 0 4281416 97428 2940348 0 0 0 4296 735 6210 14 4 79 0 3 1 0 0 4265972 97428 2954104 0 0 0 2868 691 6297 13 4 81 0 3 1 0 0 4242736 97428 2975012 0 0 0 4564 955 10067 21 8 66 0 5 1 0 0 4217148 97428 2996796 0 0 0 7040 1059 10220 23 9 61 0 8 1 0 0 4192024 97428 3019292 0 0 0 5256 1107 10753 24 8 63 0 4 1 0 0 4165704 97428 3043512 0 0 0 4240 1135 11862 26 7 66 0 1 1 0 0 4139056 97428 3067328 0 0 0 5184 1179 11436 25 9 63 0 3 1 0 0 4114124 97428 3090204 0 0 0 4200 1042 11076 23 7 64 0 6 1 0 0 4093592 97428 3103028 0 0 0 27320 1137 6365 12 15 58 11 4 1 0 0 4077564 97428 3122936 0 0 0 5000 957 9662 20 7 68 0 4 2 0 0 4056956 97428 3139108 0 0 0 22064 1426 7383 17 15 60 0 8 2 0 0 4043508 97428 3151752 0 0 0 18389 1228 5802 14 11 69 0 6 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 4032116 97428 3161168 0 0 0 48057 2334 4014 12 29 55 0 4 0 0 0 4022456 97428 3170792 0 0 0 42444 2208 3864 11 25 58 0 5 2 0 0 4012436 97428 3176364 0 0 0 35000 1310 3259 8 17 71 0 4 1 0 0 4007912 97428 3182172 0 0 0 46332 2095 3005 8 24 61 3 5 2 0 0 3999120 97428 3188468 0 0 0 47804 2062 2820 9 24 59 3 5 2 0 0 3996056 97428 3193124 0 0 0 45692 2248 2641 7 22 57 10 4 0 1 0 3986408 97428 3198288 0 0 0 48204 1703 2621 5 18 57 18 2 1 0 0 3984172 97428 3203504 0 0 0 41144 2119 2729 5 22 63 6 4 1 0 0 3979080 97428 3210696 0 0 0 39480 1582 3259 8 20 65 3 4 1 0 0 3964364 97428 3223272 0 0 0 10394 1140 5035 12 9 73 0 6 1 0 0 3940640 97428 3243824 0 0 0 5640 1014 9436 22 8 66 0 3 1 0 0 3912652 97428 3267928 0 0 0 5604 1120 11204 24 9 65 0 1 1 0 0 3875980 97428 3302288 0 0 0 10744 1571 8815 16 18 57 0 9 1 0 0 3817940 97428 3349316 0 0 0 25240 2875 2942 1 36 54 0 8 1 0 0 3755244 97428 3400224 0 0 0 31032 3200 3339 3 42 48 0 8 2 0 0 3661368 97428 3481976 0 0 0 31164 3279 3474 3 43 48 0 6 1 0 0 3602928 97428 3534648 0 0 0 33411 2293 2334 1 29 62 0 8 0 0 0 3579180 97428 3554604 0 0 0 48600 2403 2431 0 30 66 0 4 ---- _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
