On Sun, 2024-10-27 at 13:32 +0300, Konstantin Kharlamov wrote:
> On Sun, 2024-10-27 at 13:11 +0300, Konstantin Kharlamov wrote:
> > On Sat, 2024-10-26 at 23:46 -0700, Yosry Ahmed wrote:
> > > I don't think this is an edge case. I think when we swapin a page
> > > we
> > > generally leave it in the swapcache if there is no pressure on
> > > swap
> > > space. In that case the memory is not really swapped out, but
> > > because
> > > it remains in the swapcache it is still reserving a swap slot, so
> > > it
> > > shows up as swap usage.
> > >
> > > Konstantin, could you check the amount of swapcache you have,
> > > whether
> > > through /proc/vmstat or memory.stat on both user and system
> > > slices?
> >
> > Sure
> >
> > λ grep cache /sys/fs/cgroup/*/memory.stat
> > …
> > /sys/fs/cgroup/system.slice/memory.stat:swapcached
> > 434917376
> > /sys/fs/cgroup/user.slice/memory.stat:swapcached 15478784
> >
> > `434917376` is a 0.4G, not much. In comparison,
> > `system.slice/memory.swap.current` is currently `4764139520 =
> > 4.4G`.
>
> I figured since 434917376 is 10 numbers, I'd grep everything in
> memory.stat that has ten digits:
>
> λ grep -P "\d{10}$" /sys/fs/cgroup/system.slice/memory.stat
> file 2671874048
> shmem 2592768000
> zswapped 2997760000
> active_anon 1491247104
> unevictable 1269555200
>
> well, to me personally this isn't helpful, but perhaps am I missing
> something…
I found the process the "phantom memory" belongs to! I just realized
that I can see `memory.swap.current` for individual processes in a
cgroup too, and it turns out currently 4.3G belong to sddm:
/sys/fs/cgroup/system.slice/sddm.service/memory.swap.current:4723781632
systemctl confirms this:
λ systemctl status sddm
● sddm.service - Simple Desktop Display Manager
Loaded: loaded (/usr/lib/systemd/system/sddm.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-10-16 15:59:10 MSK; 1 week 3 days ago
Invocation: daadb3ed391b421b90b216122339be83
Docs: man:sddm(1)
man:sddm.conf(5)
Main PID: 720 (sddm)
Tasks: 10 (limit: 18621)
Memory: 3.3G (peak: 4.1G swap: 4.3G swap peak: 5.8G zswap: 67.6M)
CPU: 21h 30min 56.309s
CGroup: /system.slice/sddm.service
├─720 /usr/bin/sddm
└─724 /usr/lib/Xorg -nolisten tcp -background none -seat seat0 vt2 -auth /run/sddm/xauth_IKXVXT -noreset -displayfd 16
Note the `swap: 4.3G` sentence.
So, this is good news, but still doesn't answer the question where did this memory
go. Out of the 2 processes in the group, `smem` shows 2.1M for sddm and 88M for Xorg.
I even tried manually calculating:
λ sudo grep Swap /proc/72{0,4}/smaps | awk '{total+=$2} END {print "Swap memory: " total "K"}'
Swap memory: 184656K
That's 180M, for some reason very different, but whatever, still very far from 4.3G.
----------
Just to make it clear, the reason why I'm digging is that something's clearly very
wrong. And I can't blame Xorg nor sddm currently, because by all means they don't
take 4.3G of memory. The cgroup for some reason does, but the processes don't.
