Re: [LTP] Question about oom02 testcase

[Li Wang <liwang@xxxxxxxxxx>]

On Thu, Jun 1, 2023 at 5:46 PM Gou Hao <gouhao@xxxxxxxxxxxxx> wrote:

On 6/1/23 16:18, Li Wang wrote:

Hi Hao,

Thanks for reporting this, comments see below.

On Tue, May 30, 2023 at 9:26 AM Gou Hao <gouhao@xxxxxxxxxxxxx> wrote:

hello everyone,

Recently, kernel restarted while I was running oom02.
log:
```
[480156.950100] Tasks state (memory values in pages):
[480156.950101] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes
swapents oom_score_adj name
[480156.950302] [   2578]    81  2578      523        0 393216       
6          -900 dbus-daemon
[480156.950309] [   2648]   172  2596     2435        0 393216       
5             0 rtkit-daemon
[480156.950322] [   5256]     0  2826    25411        0 589824       
0             0 DetectThread
[480156.950328] [   5404]     0  5404      412        2 393216      
64         -1000 sshd
[480156.950357] [  10518]     0 10518     2586        0 393216      
10             0 at-spi2-registr
[480156.950361] [  10553]     0 10551    10543        0 458752       
9             0 QXcbEventQueue
[480156.950365] [  10867]     0 10567    17579        0 589824      
16             0 QXcbEventQueue
[480156.950370] [  10928]     0 10921     6999        0 458752      
17             0 QXcbEventQueue
[480156.950390] [  11882]     0 11811     7377        0 458752      
10             0 QXcbEventQueue
[480156.950394] [  12052]     0 12052     5823        0 458752      
21             0 fcitx
[480156.950404] [  12115]     0 12114    11678        0 524288      
21             0 QXcbEventQueue
[480156.950408] [ 101558]     0 101558     3549        0 393216       
0             0 runltp
[480156.950486] [1068864]     0 1068864      771        6 327680      
85         -1000 systemd-udevd
[480156.950552] [1035639]     0 1035639       52        0 393216      
14         -1000 oom02
[480156.950556] [1035640]     0 1035640       52        0 393216      
23         -1000 oom02
[480156.950561] [1036065]     0 1036065      493       60 393216       
0          -250 systemd-journal
[480156.950565] [1036087]     0 1036073  6258739  3543942
37814272        0             0 oom02
[480156.950572] Out of memory and no killable processes...
[480156.950575] Kernel panic - not syncing: System is deadlocked on memory
```

oom02-1036073 has been killed before crash.
log:
```
[480152.242506] [1035177]     0 1035177     4773       20 393216     
115             0 sssd_nss
[480152.242510] [1035376]     0 1035376    25500      391 589824     
602             0 tuned
[480152.242514] [1035639]     0 1035639       52        0 393216      
14         -1000 oom02
[480152.242517] [1035640]     0 1035640       52        0 393216      
19         -1000 oom02
[480152.242522] [1036065]     0 1036065      493      114 393216      
62          -250 systemd-journal
[480152.242525] [1036073]     0 1036073  6258739  3540314 37814272     
104             0 oom02
[480152.242529] Out of memory: Kill process 1036073 (oom02) score 755 or
sacrifice child
[480152.243869] Killed process 1036073 (oom02) total-vm:400559296kB,
anon-rss:226578368kB, file-rss:1728kB, shmem-rss:0kB
[480152.365804] oom_reaper: reaped process 1036073 (oom02), now
anon-rss:226594048kB, file-rss:0kB, shmem-rss:0kB
```
but its memory can not be reclaimed.I add trace-log to oom_reaper code
in kernel,
I found that there is a large range vma in the memory that cannot be
reclaimed, and the vma has the  `VM_LOCKED` flag, so cannot be reclaimed
immediately.
```log
       oom_reaper-57    [007] ....   126.063581: __oom_reap_task_mm: gh:
vma is anon:1048691, range=65536
       oom_reaper-57    [007] ....   126.063581: __oom_reap_task_mm: gh:
vma is anon:1048691, range=196608
       oom_reaper-57    [007] ....   126.063582: __oom_reap_task_mm: gh:
vma continue: 1056883, range:3221225472
       oom_reaper-57    [007] ....   126.063583: __oom_reap_task_mm: gh:
vma is anon:112, range=65536
       oom_reaper-57    [007] ....   126.063584: __oom_reap_task_mm: gh:
vma is anon:1048691, range=8388608
```
`vma continue: 1056883, range:3221225472` is the memory that can not
reclaims. 1057883(0x102073) is vma->vm_flags, it has VM_LOCKED` flag

oom02 created `nr_cpu` threads and used mmap to allocate memory. mmap
will merge continuous vma into one,
so as long as one thread is still running, the entire vma will not be
released.

In extreme cases, crashes may occur due to the lack of memory reclamation.

My question is that the crash in this case is a normal situation or a
bug (kernel or ltp bug) ?

The  ltp-oom test is originally designed to verify OOM mechanism

works for memory allocating in three types (normal, mlock, ksm)

all as expected.

Yes, your analysis is reasonable to some degree, oom_reaper
might not reclaim the VMA with locked pages  even after the

process termination.

But the exact behavior of the oom_reaper and the conditions under

which it can or cannot reclaim VMAs may vary depending on the

specific kernel version and configuration. So we shouldn't simply

regard the system panic as a Kernel or LTP defect.

And BTW, what is your tested kernel version?

hi Li Wang,

Thank you for your reply.

My kernel version is 4.19, but it's not a community version.

I have only encountered the crash once, and most of the time oom_reaper can handle it well.

I tried to find a method or flag to prevent vma merging during mmap, but couldn't find it.

That also might be related to the value of overcommit_memory,

if we set 2 (strict mode) to it, the oom_reaper can reclaim VM_LOCKED

memory more aggressively.

But in oom02 as you can see, it is set to 1 (always mode) for the

whole test, that might be the reason your system can't recover from

overcommit and finally crashed.

--

Regards,

Li Wang