On Fri, Dec 8, 2017 at 12:40 AM, Matthew Wilcox <willy@xxxxxxxxxxxxx> wrote:
> On Fri, Dec 08, 2017 at 07:30:07AM +0800, Yang Shi wrote:
>> When running stress test with KASAN enabled, the below softlockup may
>> happen occasionally:
>>
>> NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s!
>> hardirqs last enabled at (0): [< (null)>] (null)
>> hardirqs last disabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last enabled at (0): [] copy_process.part.30+0x5c6/0x1f50
>> softirqs last disabled at (0): [< (null)>] (null)
>
>> Call Trace:
>> [] __slab_free+0x19c/0x270
>> [] ___cache_free+0xa6/0xb0
>> [] qlist_free_all+0x47/0x80
>> [] quarantine_reduce+0x159/0x190
>> [] kasan_kmalloc+0xaf/0xc0
>> [] kasan_slab_alloc+0x12/0x20
>> [] kmem_cache_alloc+0xfa/0x360
>> [] ? getname_flags+0x4f/0x1f0
>> [] getname_flags+0x4f/0x1f0
>> [] getname+0x12/0x20
>> [] do_sys_open+0xf9/0x210
>> [] SyS_open+0x1e/0x20
>> [] entry_SYSCALL_64_fastpath+0x1f/0xc2
>
> This feels like papering over a problem. KASAN only calls
> quarantine_reduce() when it's allowed to block. Presumably it has
> millions of entries on the free list at this point. I think the right
> thing to do is for qlist_free_all() to call cond_resched() after freeing
> every N items.
Agree. Adding touch_softlockup_watchdog() to a random low-level
function looks like a wrong thing to do.
quarantine_reduce() already has this logic. Look at
QUARANTINE_BATCHES. It's meant to do exactly this -- limit amount of
work in quarantine_reduce() and in quarantine_remove_cache() to
reasonably-sized batches. We could simply increase number of batches
to make them smaller. But it would be good to understand what exactly
happens in this case. Batches should on a par of ~~1MB. Why freeing
1MB worth of objects (smallest of which is 32b) takes 22 seconds?
>> The code is run in irq disabled or preempt disabled context, so
>> cond_resched() can't be used in this case. Touch softlockup watchdog when
>> KASAN is enabled to suppress the warning.
>>
>> Signed-off-by: Yang Shi <yang.s@xxxxxxxxxxxxxxx>
>> ---
>> mm/slub.c | 5 +++++
>> 1 file changed, 5 insertions(+)
>>
>> diff --git a/mm/slub.c b/mm/slub.c
>> index cfd56e5..4ae435e 100644
>> --- a/mm/slub.c
>> +++ b/mm/slub.c
>> @@ -35,6 +35,7 @@
>> #include <linux/prefetch.h>
>> #include <linux/memcontrol.h>
>> #include <linux/random.h>
>> +#include <linux/nmi.h>
>>
>> #include <trace/events/kmem.h>
>>
>> @@ -2266,6 +2267,10 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
>> page->pobjects = pobjects;
>> page->next = oldpage;
>>
>> +#ifdef CONFIG_KASAN
>> + touch_softlockup_watchdog();
>> +#endif
>> +
>> } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
>> != oldpage);
>> if (unlikely(!s->cpu_partial)) {
>> --
>> 1.8.3.1
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@xxxxxxxxx. For more info on Linux MM,
>> see: http://www.linux-mm.org/ .
>> Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
