On Sun, Oct 27, 2024 at 4:28 AM Konstantin Kharlamov <Hi-Angel@xxxxxxxxx> wrote:
>
> On Sun, 2024-10-27 at 13:32 +0300, Konstantin Kharlamov wrote:
> > On Sun, 2024-10-27 at 13:11 +0300, Konstantin Kharlamov wrote:
> > > On Sat, 2024-10-26 at 23:46 -0700, Yosry Ahmed wrote:
> > > > I don't think this is an edge case. I think when we swapin a page
> > > > we
> > > > generally leave it in the swapcache if there is no pressure on
> > > > swap
> > > > space. In that case the memory is not really swapped out, but
> > > > because
> > > > it remains in the swapcache it is still reserving a swap slot, so
> > > > it
> > > > shows up as swap usage.
> > > >
> > > > Konstantin, could you check the amount of swapcache you have,
> > > > whether
> > > > through /proc/vmstat or memory.stat on both user and system
> > > > slices?
> > >
> > > Sure
> > >
> > > λ grep cache /sys/fs/cgroup/*/memory.stat
> > > …
> > > /sys/fs/cgroup/system.slice/memory.stat:swapcached
> > > 434917376
> > > /sys/fs/cgroup/user.slice/memory.stat:swapcached 15478784
> > >
> > > `434917376` is a 0.4G, not much. In comparison,
> > > `system.slice/memory.swap.current` is currently `4764139520 =
> > > 4.4G`.
> >
> > I figured since 434917376 is 10 numbers, I'd grep everything in
> > memory.stat that has ten digits:
> >
> > λ grep -P "\d{10}$" /sys/fs/cgroup/system.slice/memory.stat
> > file 2671874048
> > shmem 2592768000
> > zswapped 2997760000
> > active_anon 1491247104
> > unevictable 1269555200
> >
> > well, to me personally this isn't helpful, but perhaps am I missing
> > something…
>
> I found the process the "phantom memory" belongs to! I just realized
> that I can see `memory.swap.current` for individual processes in a
> cgroup too, and it turns out currently 4.3G belong to sddm:
>
> /sys/fs/cgroup/system.slice/sddm.service/memory.swap.current:4723781632
>
> systemctl confirms this:
>
> λ systemctl status sddm
> ● sddm.service - Simple Desktop Display Manager
> Loaded: loaded (/usr/lib/systemd/system/sddm.service; enabled; preset: disabled)
> Active: active (running) since Wed 2024-10-16 15:59:10 MSK; 1 week 3 days ago
> Invocation: daadb3ed391b421b90b216122339be83
> Docs: man:sddm(1)
> man:sddm.conf(5)
> Main PID: 720 (sddm)
> Tasks: 10 (limit: 18621)
> Memory: 3.3G (peak: 4.1G swap: 4.3G swap peak: 5.8G zswap: 67.6M)
> CPU: 21h 30min 56.309s
> CGroup: /system.slice/sddm.service
> ├─720 /usr/bin/sddm
> └─724 /usr/lib/Xorg -nolisten tcp -background none -seat seat0 vt2 -auth /run/sddm/xauth_IKXVXT -noreset -displayfd 16
>
> Note the `swap: 4.3G` sentence.
>
> So, this is good news, but still doesn't answer the question where did this memory
> go. Out of the 2 processes in the group, `smem` shows 2.1M for sddm and 88M for Xorg.
>
> I even tried manually calculating:
>
> λ sudo grep Swap /proc/72{0,4}/smaps | awk '{total+=$2} END {print "Swap memory: " total "K"}'
> Swap memory: 184656K
>
> That's 180M, for some reason very different, but whatever, still very far from 4.3G.
I think smaps will only show you swapped out mapped memory. It could be tmpfs.
One thing you can do is take a snapshot of memory.stat when
memory.swap.current is at a high value (for sddm), then swapoff, then
take another snapshot of memory.stat.
We should see an increase in either anon or shmem, which will tell us
which type of memory was swapped out.
>
> ----------
>
> Just to make it clear, the reason why I'm digging is that something's clearly very
> wrong. And I can't blame Xorg nor sddm currently, because by all means they don't
> take 4.3G of memory. The cgroup for some reason does, but the processes don't.
