On Thu, Nov 21, 2019 at 05:59:54PM +0100, Christian Borntraeger wrote: > > > On 21.11.19 17:58, Roman Gushchin wrote: > > On Thu, Nov 21, 2019 at 12:17:39PM +0100, Christian Borntraeger wrote: > >> Folks, > >> > >> I do get errors like the following when running a new testcase in our KVM CI. > >> The test basically unloads kvm, reloads with with hpage=1 (enable huge page > >> support for guests on s390) start a guest with libvirt and hugepages, shut the > >> guest down and unload the kvm module. > >> > >> WARNING: CPU: 8 PID: 208 at lib/percpu-refcount.c:108 percpu_ref_exit+0x50/0x58 > >> Modules linked in: kvm(-) xt_CHECKSUM xt_MASQUERADE bonding xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_na> > >> CPU: 8 PID: 208 Comm: kworker/8:1 Not tainted 5.2.0+ #66 > >> Hardware name: IBM 2964 NC9 712 (LPAR) > >> Workqueue: events sysfs_slab_remove_workfn > >> Krnl PSW : 0704e00180000000 0000001529746850 (percpu_ref_exit+0x50/0x58) > >> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 > >> Krnl GPRS: 00000000ffff8808 0000001529746740 000003f4e30e8e18 0036008100000000 > >> 0000001f00000000 0035008100000000 0000001fb3573ab8 0000000000000000 > >> 0000001fbdb6de00 0000000000000000 0000001529f01328 0000001fb3573b00 > >> 0000001fbb27e000 0000001fbdb69300 000003e009263d00 000003e009263cd0 > >> Krnl Code: 0000001529746842: f0a0000407fe srp 4(11,%r0),2046,0 > >> 0000001529746848: 47000700 bc 0,1792 > >> #000000152974684c: a7f40001 brc 15,152974684e > >> >0000001529746850: a7f4fff2 brc 15,1529746834 > >> 0000001529746854: 0707 bcr 0,%r7 > >> 0000001529746856: 0707 bcr 0,%r7 > >> 0000001529746858: eb8ff0580024 stmg %r8,%r15,88(%r15) > >> 000000152974685e: a738ffff lhi %r3,-1 > >> Call Trace: > >> ([<000003e009263d00>] 0x3e009263d00) > >> [<00000015293252ea>] slab_kmem_cache_release+0x3a/0x70 > >> [<0000001529b04882>] kobject_put+0xaa/0xe8 > >> [<000000152918cf28>] process_one_work+0x1e8/0x428 > >> [<000000152918d1b0>] worker_thread+0x48/0x460 > >> [<00000015291942c6>] kthread+0x126/0x160 > >> [<0000001529b22344>] ret_from_fork+0x28/0x30 > >> [<0000001529b2234c>] kernel_thread_starter+0x0/0x10 > >> Last Breaking-Event-Address: > >> [<000000152974684c>] percpu_ref_exit+0x4c/0x58 > >> ---[ end trace b035e7da5788eb09 ]--- > >> > >> I have bisected this to > >> # first bad commit: [f0a3a24b532d9a7e56a33c5112b2a212ed6ec580] mm: memcg/slab: rework non-root kmem_cache lifecycle management > >> > >> unmounting /sys/fs/cgroup/memory/ before the test makes the problem go away so > >> it really seems to be related to the new percpu-refs from this patch. > >> > >> Any ideas? > > > > Hello, Christian! > > > > It seems to be a race between releasing of the root kmem_cache (caused by rmmod) > > and a memcg kmem_cache. Does delaying rmmod for say, a minute, "resolve" the > > issue? > > Yes, rmmod has to be called directly after the guest shutdown to see the issue. > See my 2nd mail. I see. Do you know, which kmem_cache it is? If not, can you, please, figure it out? I tried to reproduce the issue, but wasn't successful so far. So I wonder what can make your case special. Thanks!
