On Sat, Oct 26, 2024 at 5:29 PM Konstantin Kharlamov <Hi-Angel@xxxxxxxxx> wrote: > > That was a good idea! The > `/sys/fs/cgroup/system.slice/memory.swap.current` seems to have the > missing half of the SWAP memory. From my understanding of the > `systemctl status` graph `sytem.slice` and `user.slice` groups do not > intersect, and by adding up `system.slice/…` + `user.slice/…` I get > around 8G. > > However, I'm still unclear what does this memory belong to. > `system.slice/memory.swap.current` is 4.4G currently, that's a lot and > I'm not seeing anything that could take so much memory. I assume you do not have any proactive memory reclaimer? :) I believe the top utility can display swap usage by process. Have you tried that? There are a couple of edge cases - for instance, if you disable zswap writeback and zswap at the same time. We will allocate slots on swapfile, and store it at the page table entry, but we cannot store the page's content in zswap or the swapfile, so the page remains in memory. You're occupying swap space, but are not really saving any memory usage. IIRC, there is also an edge case where a page is faulted back into memory from swap, but the associated swap space cannot be immediately released. This should be temporary though - memory reclaimer will attempt to release these pages later on, or they can be released when we scan the swapfile for slots during swap out. > > An even larger related mystery is why does this memory not show up in > `smem` numbers for individual applications (which calculates it by > going over `/proc/$pid/smaps` for every pid). > > > The other possibility is that the pages are swapped out from the root > > cgroup, in which case they won't show up in memory.swap.current as > > they are basically unaccounted. Although typically user processes > > should not be running in the root cgroup. > > > > > "phantom swap memory" is hidden in `user.slice`, because if I wait > > > till > > > OOM-killer gets triggered and kills some app, my user-systemd gets > > > crashed for some reason, taking down the entire user session, and > > > afterwards SWAP is almost free. > > > > Did you check the OOM logs? It is possible that the OOM killer kills > > some system process that has some memory in swap as well. > > I did, logs are pretty uninteresting. OOM kills `electron` (of element- > desktop), but I tried closing it before the OOM, that didn't have much > influence. Just an arbitrary victim. Then a few lines later a `Process > 560296 (systemd) of user 1000 terminated abnormally with signal > 11/SEGV`. Wasn't able to get stacktrace for systemd with Archlinux's > debuginfo servers. And then everything gets down with systemd. > > I just tried closing every application I have open and I still got 5.5 > in SWAP. Well, obviously there are services still running, Plasma, > i3wm… Not many suspects left though. This beats me. I don't know the process situation in your laptop. Sorry :)
