On Thu, Jul 02, 2020 at 12:22:03PM +0530, Naresh Kamboju wrote: > On Thu, 2 Jul 2020 at 00:16, Roman Gushchin <guro@xxxxxx> wrote: > > > > On Wed, Jul 01, 2020 at 10:29:04AM +0200, Michal Hocko wrote: > > > Smells like a different observable problem with the same/similar culprit > > > as http://lkml.kernel.org/r/CA+G9fYtrgF_EZHi0vi+HyWiXT5LGggDhVXtNspc=OzzFhL=xRQ@xxxxxxxxxxxxxx > > > > > > On Wed 01-07-20 13:48:57, Naresh Kamboju wrote: > > > > While running LTP mm test suite on x86_64 device the BUG: Bad page > > > > state in process > > > > noticed on linux-next 20200630 tag. > > > > > > > > Steps to reproduce: > > > > - boot linux-next 20200630 kernel on x86_64 device > > > > - cd /opt/ltp > > > > - ./runltp -f mm > > > > > > > > metadata: > > > > git branch: master > > > > git repo: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git > > > > git commit: f2b92b14533e646e434523abdbafddb727c23898 > > > > git describe: next-20200630 > > > > kernel-config: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__builds.tuxbuild.com_j60yrp7CUpq3LCmqMB8Wdg_kernel.config&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=h_KJ0e7abuh0BK2eDlDmWnAxqHPccpqchPgBS-oJcVE&s=qofg2XRToTeHvi8vSdOvDPtKpJsUqf3IWfqwieZqITg&e= > > > > > > > > Test crash dump: > > > > [ 803.905169] Node 0 Normal: 2608*4kB (UMEH) 1380*8kB (UMEH) 64*16kB > > > > (MEH) 28*32kB (MEH) 13*64kB (UMEH) 164*128kB (UMEH) 39*256kB (UE) > > > > 1*512kB (M) 1*1024kB (M) 1*2048kB (M) 1*4096kB (M) = 62880kB > > > > [ 803.922375] Node 0 hugepages_total=0 hugepages_free=0 > > > > hugepages_surp=0 hugepages_size=2048kB > > > > [ 803.930806] 2418 total pagecache pages > > > > [ 803.934559] 0 pages in swap cache > > > > [ 803.937878] Swap cache stats: add 0, delete 0, find 0/0 > > > > [ 803.943108] Free swap = 0kB > > > > [ 803.945997] Total swap = 0kB > > > > [ 803.948885] 4181245 pages RAM > > > > [ 803.951857] 0 pages HighMem/MovableOnly > > > > [ 803.955695] 626062 pages reserved > > > > [ 803.959016] Tasks state (memory values in pages): > > > > [ 803.963722] [ pid ] uid tgid total_vm rss pgtables_bytes > > > > swapents oom_score_adj name > > > > [ 803.972336] [ 332] 0 332 8529 507 106496 > > > > 0 0 systemd-journal > > > > [ 803.981387] [ 349] 0 349 10730 508 118784 > > > > 0 -1000 systemd-udevd > > > > [ 803.990262] [ 371] 993 371 8666 108 118784 > > > > 0 0 systemd-network > > > > [ 803.999306] [ 379] 992 379 9529 99 110592 > > > > 0 0 systemd-resolve > > > > [ 804.008347] [ 388] 0 388 2112 19 61440 > > > > 0 0 syslogd > > > > [ 804.016709] [ 389] 995 389 9308 108 122880 > > > > 0 0 avahi-daemon > > > > [ 804.025517] [ 391] 0 391 1075 21 57344 > > > > 0 0 acpid > > > > [ 804.033695] [ 394] 995 394 9277 68 114688 > > > > 0 0 avahi-daemon > > > > [ 804.042476] [ 396] 996 396 7241 154 102400 > > > > 0 -900 dbus-daemon > > > > [ 804.051170] [ 397] 0 397 2313 72 65536 > > > > 0 0 crond > > > > [ 804.059349] [ 399] 0 399 34025 161 167936 > > > > 0 0 thermald > > > > [ 804.067783] [ 400] 0 400 8615 115 110592 > > > > 0 0 systemd-logind > > > > [ 804.076734] [ 401] 0 401 2112 32 57344 > > > > 0 0 klogd > > > > [ 804.084907] [ 449] 65534 449 3245 39 69632 > > > > 0 0 dnsmasq > > > > [ 804.093254] [ 450] 0 450 3187 33 73728 > > > > 0 0 agetty > > > > [ 804.101541] [ 452] 0 452 3187 33 73728 > > > > 0 0 agetty > > > > [ 804.109826] [ 453] 0 453 14707 107 159744 > > > > 0 0 login > > > > [ 804.118007] [ 463] 0 463 9532 163 122880 > > > > 0 0 systemd > > > > [ 804.126362] [ 464] 0 464 16132 424 172032 > > > > 0 0 (sd-pam) > > > > [ 804.134803] [ 468] 0 468 4538 105 81920 > > > > 0 0 sh > > > > [ 804.142741] [ 472] 0 472 11102 83 131072 > > > > 0 0 su > > > > [ 804.150680] [ 473] 0 473 4538 99 81920 > > > > 0 0 sh > > > > [ 804.158637] [ 519] 0 519 2396 57 61440 > > > > 0 0 lava-test-runne > > > > [ 804.167700] [ 1220] 0 1220 2396 52 61440 > > > > 0 0 lava-test-shell > > > > [ 804.176738] [ 1221] 0 1221 2396 55 61440 > > > > 0 0 sh > > > > [ 804.184680] [ 1223] 0 1223 2462 135 61440 > > > > 0 0 ltp.sh > > > > [ 804.192946] [ 1242] 0 1242 2462 134 61440 > > > > 0 0 ltp.sh > > > > [ 804.201207] [ 1243] 0 1243 2462 134 61440 > > > > 0 0 ltp.sh > > > > [ 804.209475] [ 1244] 0 1244 2462 134 61440 > > > > 0 0 ltp.sh > > > > [ 804.217742] [ 1245] 0 1245 2561 229 65536 > > > > 0 0 runltp > > > > [ 804.226010] [ 1246] 0 1246 1072 15 53248 > > > > 0 0 tee > > > > [ 804.234012] [ 1313] 0 1313 1070 29 53248 > > > > 0 0 ltp-pan > > > > [ 804.242374] [ 3216] 0 3216 1613 20 53248 > > > > 0 0 oom01 > > > > [ 804.250554] [ 3217] 0 3217 1646 31 57344 > > > > 0 0 oom01 > > > > [ 804.258728] [ 3245] 0 3245 81271 469 266240 > > > > 0 0 NetworkManager > > > > [ 804.267688] [ 3249] 0 3249 6422 54 98304 > > > > 0 0 systemd-hostnam > > > > [ 804.276734] [ 3250] 0 3250 52976 178 172032 > > > > 0 0 nm-dispatcher > > > > [ 804.285603] [ 3254] 998 3254 131113 828 245760 > > > > 0 0 polkitd > > > > [ 804.293956] [ 3261] 0 3261 4726385 3349389 26939392 > > > > 0 0 oom01 > > > > [ 804.302129] [ 3265] 0 3265 3187 33 73728 > > > > 0 0 agetty > > > > [ 804.310397] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=oom01,pid=3261,uid=0 > > > > [ 804.322751] Out of memory: Killed process 3261 (oom01) > > > > total-vm:18905540kB, anon-rss:13397556kB, file-rss:0kB, shmem-rss:0kB, > > > > UID:0 pgtables:26308kB oom_score_adj:0 > > > > [ 806.652952] oom_reaper: reaped process 3261 (oom01), now > > > > anon-rss:0kB, file-rss:0kB, shmem-rss:0kB > > > > [ 807.579373] BUG: Bad page state in process kworker/u8:12 pfn:374308 > > > > [ 807.579521] BUG: Bad page state in process kworker/u8:13 pfn:4182a4 > > > > [ 807.585734] page:ffffea000dd0c200 refcount:0 mapcount:0 > > > > mapping:0000000000000000 index:0xffff88837430e000 > > > > head:ffffea000dd0c200 order:3 compound_mapcount:0 compound_pincount:0 > > > > [ 807.585736] flags: 0x200000000010000(head) > > > > [ 807.585740] raw: 0200000000010000 ffffea000dce6e00 0000000200000002 > > > > 0000000000000000 > > > > [ 807.592099] page:ffffea001060a900 refcount:0 mapcount:0 > > > > mapping:0000000000000000 index:0xffff8884182a5e00 > > > > head:ffffea001060a900 order:1 compound_mapcount:0 > > > > [ 807.607719] raw: ffff88837430e000 0000000000040000 00000000ffffffff > > > > ffff8883bda6cac1 > > > > [ 807.607720] page dumped because: page still charged to cgroup > > > > [ 807.607720] page->mem_cgroup:ffff8883bda6cac1 > > > > [ 807.607721] Modules linked in: x86_pkg_temp_thermal > > > > [ 807.607725] CPU: 0 PID: 3242 Comm: kworker/u8:12 Not tainted > > > > 5.8.0-rc3-next-20200630 #1 > > > > [ 807.607727] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS > > > > 2.0b 07/27/2017 > > > > [ 807.607731] Workqueue: rpciod rpc_async_schedule > > > > [ 807.611836] flags: 0x200000000010000(head) > > > > [ 807.619563] Call Trace: > > > > [ 807.619567] dump_stack+0x84/0xba > > > > [ 807.619569] bad_page.cold+0x7b/0xac > > > > [ 807.619573] __free_pages_ok+0x95b/0xab0 > > > > [ 807.633461] raw: 0200000000010000 dead000000000100 dead000000000122 > > > > 0000000000000000 > > > > [ 807.641189] __free_pages+0x42/0x50 > > > > [ 807.641191] __free_slab+0xcd/0x1f0 > > > > Hm, interesting, it means that page->obj_cgroups is still set. > > But before __free_pages() __free_slab() always calls uncharge_slab_page(), > > which sets page->obj_cgroups to NULL except when !memcg_kmem_enabled(). > > > > So it makes me think that somehow memcg_kmem_enabled() became false > > after being true, which can cause refcounting problems as well. > > > > Naresh, can you, please, check if the following patch solves problems? > > And thank you for reporting the problem! > > > > Thanks for providing a quick fix patch. > > > > > From c97afecd32c0db5e024be9ba72f43d22974f5bcd Mon Sep 17 00:00:00 2001 > > From: Roman Gushchin <guro@xxxxxx> > > Date: Wed, 1 Jul 2020 11:05:32 -0700 > > Subject: [PATCH] mm: kmem: make memcg_kmem_enabled() irreversible > > > > Historically the kernel memory accounting was an opt-in feature, which > > could be enabled for individual cgroups. But now it's not true, and > > it's on by default both on cgroup v1 and cgroup v2. And as long as a > > user has at least one non-root memory cgroup, the kernel memory > > accounting is on. So in most setups it's either always on (if memory > > cgroups are in use and kmem accounting is not disabled), either always > > off (otherwise). > > > > memcg_kmem_enabled() is used in many places to guard the kernel memory > > accounting code. If memcg_kmem_enabled() can reverse from returning > > true to returning false (as now), we can't rely on it on release paths > > and have to check if it was on before. > > > > If we'll make memcg_kmem_enabled() irreversible (always returning true > > after returning it for the first time), it'll make the general logic > > more simple and robust. It also will allow to guard some checks which > > otherwise would stay unguarded. > > > > Signed-off-by: Roman Gushchin <guro@xxxxxx> > Reported-by: Naresh Kamboju <naresh.kamboju@xxxxxxxxxx> > Tested-by: Naresh Kamboju <naresh.kamboju@xxxxxxxxxx> > > The below patch applied on top of linux-next 20200701and tested and > the reported bug got fixed on arm64, arm, x86_64 and i386. Hello, Naresh! Thank you very much for testing the patch! Btw, did it also fix the problem reported here: https://lore.kernel.org/linux-mm/20200701081951.GL2369@xxxxxxxxxxxxxx/T/#t ? Thanks! Roman
