On Sun, 2024-10-27 at 12:31 -0700, Yosry Ahmed wrote:
> On Sun, Oct 27, 2024 at 4:28 AM Konstantin Kharlamov
> <Hi-Angel@xxxxxxxxx> wrote:
> >
> > On Sun, 2024-10-27 at 13:32 +0300, Konstantin Kharlamov wrote:
> > > On Sun, 2024-10-27 at 13:11 +0300, Konstantin Kharlamov wrote:
> > > > On Sat, 2024-10-26 at 23:46 -0700, Yosry Ahmed wrote:
> > > > > I don't think this is an edge case. I think when we swapin a
> > > > > page
> > > > > we
> > > > > generally leave it in the swapcache if there is no pressure
> > > > > on
> > > > > swap
> > > > > space. In that case the memory is not really swapped out, but
> > > > > because
> > > > > it remains in the swapcache it is still reserving a swap
> > > > > slot, so
> > > > > it
> > > > > shows up as swap usage.
> > > > >
> > > > > Konstantin, could you check the amount of swapcache you have,
> > > > > whether
> > > > > through /proc/vmstat or memory.stat on both user and system
> > > > > slices?
> > > >
> > > > Sure
> > > >
> > > > λ grep cache /sys/fs/cgroup/*/memory.stat
> > > > …
> > > > /sys/fs/cgroup/system.slice/memory.stat:swapcached
> > > > 434917376
> > > > /sys/fs/cgroup/user.slice/memory.stat:swapcached 15478784
> > > >
> > > > `434917376` is a 0.4G, not much. In comparison,
> > > > `system.slice/memory.swap.current` is currently `4764139520 =
> > > > 4.4G`.
> > >
> > > I figured since 434917376 is 10 numbers, I'd grep everything in
> > > memory.stat that has ten digits:
> > >
> > > λ grep -P "\d{10}$" /sys/fs/cgroup/system.slice/memory.stat
> > > file 2671874048
> > > shmem 2592768000
> > > zswapped 2997760000
> > > active_anon 1491247104
> > > unevictable 1269555200
> > >
> > > well, to me personally this isn't helpful, but perhaps am I
> > > missing
> > > something…
> >
> > I found the process the "phantom memory" belongs to! I just
> > realized
> > that I can see `memory.swap.current` for individual processes in a
> > cgroup too, and it turns out currently 4.3G belong to sddm:
> >
> >
> > /sys/fs/cgroup/system.slice/sddm.service/memory.swap.current:472378
> > 1632
> >
> > systemctl confirms this:
> >
> > λ systemctl status sddm
> > ● sddm.service - Simple Desktop Display Manager
> > Loaded: loaded (/usr/lib/systemd/system/sddm.service;
> > enabled; preset: disabled)
> > Active: active (running) since Wed 2024-10-16 15:59:10 MSK;
> > 1 week 3 days ago
> > Invocation: daadb3ed391b421b90b216122339be83
> > Docs: man:sddm(1)
> > man:sddm.conf(5)
> > Main PID: 720 (sddm)
> > Tasks: 10 (limit: 18621)
> > Memory: 3.3G (peak: 4.1G swap: 4.3G swap peak: 5.8G zswap:
> > 67.6M)
> > CPU: 21h 30min 56.309s
> > CGroup: /system.slice/sddm.service
> > ├─720 /usr/bin/sddm
> > └─724 /usr/lib/Xorg -nolisten tcp -background none -
> > seat seat0 vt2 -auth /run/sddm/xauth_IKXVXT -noreset -displayfd 16
> >
> > Note the `swap: 4.3G` sentence.
> >
> > So, this is good news, but still doesn't answer the question where
> > did this memory
> > go. Out of the 2 processes in the group, `smem` shows 2.1M for sddm
> > and 88M for Xorg.
> >
> > I even tried manually calculating:
> >
> > λ sudo grep Swap /proc/72{0,4}/smaps | awk '{total+=$2} END
> > {print "Swap memory: " total "K"}'
> > Swap memory: 184656K
> >
> > That's 180M, for some reason very different, but whatever, still
> > very far from 4.3G.
FTR, the reason I got "very different 180M" is I by mistake added up
SwapPSS as well.
> I think smaps will only show you swapped out mapped memory. It could
> be tmpfs.
>
> One thing you can do is take a snapshot of memory.stat when
> memory.swap.current is at a high value (for sddm), then swapoff, then
> take another snapshot of memory.stat.
>
> We should see an increase in either anon or shmem, which will tell us
> which type of memory was swapped out.
Okay. I will have to wait, because the session got killed by OOM. But I
think it's gonna reproduce in just a few days, my new workflow seems to
be triggering that a lot.
I took this chance to rename the thread as well, otherwise I'm gonna
forget it upon writing the next email.
