Hi everyone,
I've been facing this problem for the past few weeks and I'm out of
ideas of how to analyze it, maybe someone here has some thoughts on
how to address it.
Basically, if I compare the output of `free` and the total RSS
reported by `ps` there are 200 GiB of difference:
$ free
total used free shared buff/cache available
Mem: 521933440 371541084 19246028 4100 131146328 117933472
Swap: 20971516 15208556 5762960
$ ps -eo rss | awk '{ rss += $0 } END { print rss}'
176765072
total used: 386749640
total RSS: 176765072
difference: 209984568 (~200.3 GiB)
I have some programs running that make intense use of networking with
thousands of sockets opened, so I checked sockstat but with no luck:
$ cat /proc/net/sockstat
sockets: used 90319
TCP: inuse 40798 orphan 0 tw 56 alloc 89689 mem 558949 # *4 = 2235796
KiB = ~2.13 GiB
UDP: inuse 31 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
Even /proc/meminfo is quite useless [1].
I started to kill one process at a time till I found the culprit: a
Java process with high network usage. But inspecting this process'
memory consumption didn't give me any conclusion because its memory
consumption appears to be low:
$ sudo pmap -x 268169 | head -n2 | tail -n1; sudo pmap -x 268169 | tail -n 1
Address Kbytes RSS Dirty Mode Mapping
total kB 171403084 23775460 23697900
~ 22.7 GiB
But this is the process that is helding the resources because, once
killed, the memory difference/leak disappears.
How is it possible that no utility or kernel information is capable of
outlining this memory leak? What can I possibly do to investigate it
further? For sure there's a problem releasing resources in the Java
code, but I would like to see it at a kernel level.
Any help is very appreciated.
Thank you in advance,
Andrea
[1] $ cat /proc/meminfo
MemTotal: 521933440 kB
MemFree: 20600964 kB
MemAvailable: 118570008 kB
Buffers: 18092 kB
Cached: 125086816 kB
SwapCached: 1792868 kB
Active: 115786784 kB
Inactive: 187218964 kB
Active(anon): 79144440 kB
Inactive(anon): 98768932 kB
Active(file): 36642344 kB
Inactive(file): 88450032 kB
Unevictable: 20556 kB
Mlocked: 20556 kB
SwapTotal: 20971516 kB
SwapFree: 5762448 kB
Dirty: 114392 kB
Writeback: 0 kB
AnonPages: 176129992 kB
Mapped: 315908 kB
Shmem: 4100 kB
KReclaimable: 5322088 kB
Slab: 13095480 kB
SReclaimable: 5322088 kB
SUnreclaim: 7773392 kB
KernelStack: 135600 kB
PageTables: 496504 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 542904956 kB
Committed_AS: 210582988 kB
VmallocTotal: 133143592960 kB
VmallocUsed: 256200 kB
VmallocChunk: 0 kB
Percpu: 62976 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 32768 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@xxxxxxxxxxxxxxxxx
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
