On 08/01/2021 14:41, Michal Hocko wrote:
> On Wed 06-01-21 16:20:15, Milan Broz wrote:
>> Hi,
>>
>> we use mlockall(MCL_CURRENT | MCL_FUTURE) / munlockall() in cryptsetup code
>> and someone tried to use it with hardened memory allocator library.
>>
>> Execution time was increased to extreme (minutes) and as we found, the problem
>> is in munlockall().
>>
>> Here is a plain reproducer for the core without any external code - it takes
>> unlocking on Fedora rawhide kernel more than 30 seconds!
>> I can reproduce it on 5.10 kernels and Linus' git.
>>
>> The reproducer below tries to mmap large amount memory with PROT_NONE (later never used).
>> The real code of course does something more useful but the problem is the same.
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <fcntl.h>
>> #include <sys/mman.h>
>>
>> int main (int argc, char *argv[])
>> {
>> void *p = mmap(NULL, 1UL << 41, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>>
>> if (p == MAP_FAILED) return 1;
>>
>> if (mlockall(MCL_CURRENT | MCL_FUTURE)) return 1;
>> printf("locked\n");
>>
>> if (munlockall()) return 1;
>> printf("unlocked\n");
>>
>> return 0;
>> }
>>
>> In traceback I see that time is spent in munlock_vma_pages_range.
>>
>> [ 2962.006813] Call Trace:
>> [ 2962.006814] ? munlock_vma_pages_range+0xe7/0x4b0
>> [ 2962.006814] ? vma_merge+0xf3/0x3c0
>> [ 2962.006815] ? mlock_fixup+0x111/0x190
>> [ 2962.006815] ? apply_mlockall_flags+0xa7/0x110
>> [ 2962.006816] ? __do_sys_munlockall+0x2e/0x60
>> [ 2962.006816] ? do_syscall_64+0x33/0x40
>> ...
>>
>> Or with perf, I see
>>
>> # Overhead Command Shared Object Symbol
>> # ........ ....... ................. .....................................
>> #
>> 48.18% lock [kernel.kallsyms] [k] lock_is_held_type
>> 11.67% lock [kernel.kallsyms] [k] ___might_sleep
>> 10.65% lock [kernel.kallsyms] [k] follow_page_mask
>> 9.17% lock [kernel.kallsyms] [k] debug_lockdep_rcu_enabled
>> 6.73% lock [kernel.kallsyms] [k] munlock_vma_pages_range
>> ...
>>
>>
>> Could please anyone check what's wrong here with the memory locking code?
>> Running it on my notebook I can effectively DoS the system :)
>>
>> Original report is https://gitlab.com/cryptsetup/cryptsetup/-/issues/617
>> but this is apparently a kernel issue, just amplified by usage of munlockall().
>
> Which kernel version do you see this with? Have older releases worked
> better?
Hi,
I tried 5.10 stable and randomly few kernels I have built on testing VM (5.3 was the oldest),
it seems to be very similar run time, so the problem is apparently old...(I can test some specific kernel version if it make any sense).
For mainline (reproducer above):
With 5.11.0-0.rc2.20210106git36bbbd0e234d.117.fc34.x86_64 (latest Fedora rawhide kernel build - many debug options are on)
# time ./lock
locked
unlocked
real 0m32.287s
user 0m0.001s
sys 0m32.126s
Today's Linus git - 5.11.0-rc2+ in my testing x86_64 VM (no extensive kernel debug options):
# time ./lock
locked
unlocked
real 0m4.172s
user 0m0.000s
sys 0m4.172s
m.
