Big kernel dynamic memory usage (non-cache)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi!

In short, how to get detailed information what is the "noncache" column in
"kernel dynamic memory" in smem -w output?

# smem -wtk
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory          1.5G     637.7M     853.5M 
userspace memory               2.0G     142.7M       1.9G 
free memory                  383.5M     383.5M          0 
----------------------------------------------------------
                               3.9G       1.1G       2.7G

Little background info:

I'm experiencing a weird problem with the kernel, where it does allocate
lots of memory, but I can't find what it is used for.
After the system has been running for few days (around a week usually) it
starts to swap quite heavily (if I don't close the biggest applications 
quickly to recover I usually have to reset it as even remote ssh access 
doesn't work, it responds to pings so it's not a hard lock up, Magic SysRq
doesn't restart it though). Thing is, it's enough to log-out from the X
environment and back in to make it go back to normal (where the noncache
memory is less than 200MiB).

It's a workstation, with KMail, Opera and the usual background KDE services 
running in the back (with file indexing turned off). Any other processes 
are either short lived or not using any big amounts (>5MiB) of memory.
Archlinux, so the packages (both kernel and userspace) are either vanilla or
with minimal patches.

The sum of USS of all processes in the system, as reported by smem 
(the sum of RSS from `ps` is only 10-15% larger), during all that time oscillates 
around 2GiB and basically never reaches 3GiB. I have tried running with zram 
and it made the situation a bit better, but it sometimes still locks up 
(because it starts swapping so hard).

As restarting X helps, it led me to believe it may be a regression from this
bug: https://bugzilla.redhat.com/show_bug.cgi?id=615505
but it doesn't look like it to me:

# cat /sys/kernel/debug/dri/0/gem_names 
  name     size handles refcount
     1  7680000       1        3
     2 15564800       1        3

Some statistics and information on the system follow.

Regards,
Hubert Kario

# uname -a
Linux bursa22 3.6.11-1-ARCH #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012 x86_64 GNU/Linux

# free -k
             total       used       free     shared    buffers     cached
Mem:       4051956    3598176     453780          0      30264     646828
-/+ buffers/cache:    2921084    1130872
Swap:      4051952     292524    3759428

# lsmod 
Module                  Size  Used by
vboxdrv              1823470  0 
iptable_filter          1457  1 
iptable_mangle          1585  1 
xt_tcpudp               2472  14 
xt_connmark             1910  12 
xt_mark                 1214  13 
iptable_nat             4025  1 
nf_nat                 15371  1 iptable_nat
nf_conntrack_ipv4       7799  15 nf_nat,iptable_nat
nf_defrag_ipv4          1340  1 nf_conntrack_ipv4
nf_conntrack           64101  4 nf_nat,xt_connmark,iptable_nat,nf_conntrack_ipv4
ip_tables              16947  3 iptable_filter,iptable_mangle,iptable_nat
x_tables               17000  7 xt_mark,ip_tables,xt_tcpudp,iptable_filter,xt_connmark,iptable_mangle,iptable_nat
tun                    16593  2 
zram                    9460  2 
xfs                   748208  1 
snd_hda_codec_hdmi     24529  1 
fuse                   69213  5 
btrfs                 718583  2 
coretemp                6071  0 
kvm_intel             124718  0 
crc32c                  1769  1 
libcrc32c               1003  1 btrfs
zlib_deflate           20437  1 btrfs
kvm                   374014  1 kvm_intel
radeon                859687  2 
iTCO_wdt                5256  0 
iTCO_vendor_support     1930  1 iTCO_wdt
gpio_ich                4064  0 
snd_hda_codec_realtek    61420  1 
ttm                    64385  1 radeon
drm_kms_helper         32878  1 radeon
drm                   218903  4 ttm,drm_kms_helper,radeon
r8169                  56872  0 
i2c_algo_bit            5392  1 radeon
snd_hda_intel          26181  4 
mii                     4092  1 r8169
snd_hda_codec          98034  3 snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_intel
snd_hwdep               6429  1 snd_hda_codec
snd_pcm                75735  3 snd_hda_codec_hdmi,snd_hda_codec,snd_hda_intel
snd_page_alloc          7218  2 snd_pcm,snd_hda_intel
snd_timer              18935  1 snd_pcm
acpi_cpufreq            5934  0 
snd                    60189  15 snd_hda_codec_realtek,snd_hwdep,snd_timer,snd_hda_codec_hdmi,snd_pcm,snd_hda_codec,snd_hda_intel
mperf                   1300  1 acpi_cpufreq
intel_agp              10745  0 
i2c_i801                9572  0 
soundcore               5443  1 snd
intel_gtt              15660  1 intel_agp
microcode              12346  0 
i2c_core               20708  5 drm,i2c_i801,drm_kms_helper,i2c_algo_bit,radeon
lpc_ich                10610  0 
pcspkr                  1900  0 
processor              26856  3 acpi_cpufreq
button                  4663  0 
evdev                  10267  7 
ext4                  440435  4 
crc16                   1360  1 ext4
jbd2                   78802  1 ext4
mbcache                 6027  1 ext4
dm_mod                 72106  25 
raid456                55308  1 
async_raid6_recov       5375  1 raid456
async_memcpy            1367  2 raid456,async_raid6_recov
async_pq                3261  2 raid456,async_raid6_recov
async_xor               2394  3 async_pq,raid456,async_raid6_recov
xor                     8866  1 async_xor
async_tx                1903  5 async_pq,raid456,async_xor,async_memcpy,async_raid6_recov
raid6_pq               86983  2 async_pq,async_raid6_recov
raid10                 40365  1 
md_mod                104017  3 raid456,raid10
hid_generic             1114  0 
usbhid                 37036  0 
hid                    85974  2 hid_generic,usbhid
sr_mod                 14824  0 
cdrom                  35521  1 sr_mod
sd_mod                 29560  27 
ata_generic             3339  0 
uhci_hcd               23437  0 
pata_acpi               3452  0 
sata_via                8124  5 
ahci                   21361  12 
libahci                20024  1 ahci
pata_it8213             3587  1 
firewire_ohci          31942  0 
libata                167757  6 ahci,pata_acpi,sata_via,libahci,ata_generic,pata_it8213
scsi_mod              133434  3 libata,sd_mod,sr_mod
firewire_core          53174  1 firewire_ohci
crc_itu_t               1364  1 firewire_core
ehci_hcd               41849  0 
usbcore               150472  3 uhci_hcd,ehci_hcd,usbhid
usb_common               955  1 usbcore

# smem -kt | sed -n '1p;$p'
  PID User     Command                         Swap      USS      PSS      RSS 
  110 6                                      281.6M     2.0G     2.0G     2.3G

# cat /proc/meminfo 
MemTotal:        4051956 kB
MemFree:          452512 kB
Buffers:           29784 kB
Cached:           648796 kB
SwapCached:       144308 kB
Active:          1555176 kB
Inactive:        1175416 kB
Active(anon):    1238968 kB
Inactive(anon):   868836 kB
Active(file):     316208 kB
Inactive(file):   306580 kB
Unevictable:        7720 kB
Mlocked:            7720 kB
SwapTotal:       4051952 kB
SwapFree:        3759416 kB
Dirty:               372 kB
Writeback:             0 kB
AnonPages:       1982576 kB
Mapped:           148724 kB
Shmem:             52248 kB
Slab:             185580 kB
SReclaimable:      69316 kB
SUnreclaim:       116264 kB
KernelStack:        3320 kB
PageTables:        43996 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6077928 kB
Committed_AS:    3751688 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      139780 kB
VmallocChunk:   34359570976 kB
HardwareCorrupted:     0 kB
AnonHugePages:    442368 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     2536128 kB
DirectMap2M:     1656832 kB

# cat /proc/vmstat 
nr_free_pages 113256
nr_inactive_anon 217210
nr_active_anon 309694
nr_inactive_file 76546
nr_active_file 79049
nr_unevictable 1930
nr_mlock 1930
nr_anon_pages 385048
nr_mapped 37179
nr_file_pages 205623
nr_dirty 24
nr_writeback 0
nr_slab_reclaimable 17329
nr_slab_unreclaimable 29066
nr_page_table_pages 10973
nr_kernel_stack 413
nr_unstable 0
nr_bounce 0
nr_vmscan_write 3103053
nr_vmscan_immediate_reclaim 21226934
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 13062
nr_dirtied 39768555
nr_written 38668348
numa_hit 1043570022
numa_miss 0
numa_foreign 0
numa_interleave 7629
numa_local 1043570022
numa_other 0
nr_anon_transparent_hugepages 216
nr_dirty_threshold 76584
nr_dirty_background_threshold 38292
pgpgin 538657436
pgpgout 172521740
pswpin 350320
pswpout 404062
pgalloc_dma 2
pgalloc_dma32 666912295
pgalloc_normal 571432892
pgalloc_movable 0
pgfree 1243823469
pgactivate 73410706
pgdeactivate 72896276
pgfault 544723643
pgmajfault 2023827
pgrefill_dma 0
pgrefill_dma32 64507672
pgrefill_normal 18650343
pgrefill_movable 0
pgsteal_kswapd_dma 0
pgsteal_kswapd_dma32 3005794
pgsteal_kswapd_normal 26891312
pgsteal_kswapd_movable 0
pgsteal_direct_dma 0
pgsteal_direct_dma32 15653023
pgsteal_direct_normal 42635574
pgsteal_direct_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 3136881
pgscan_kswapd_normal 32518914
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 361983838
pgscan_direct_normal 6556538987
pgscan_direct_movable 0
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 24512
slabs_scanned 35039616
kswapd_inodesteal 193648
kswapd_low_wmark_hit_quickly 654504
kswapd_high_wmark_hit_quickly 690974
kswapd_skip_congestion_wait 2977
pageoutrun 645618572
allocstall 7647697
pgrotated 285215
compact_blocks_moved 23846881
compact_pages_moved 4897380
compact_pagemigrate_failed 48162348
compact_stall 145117
compact_fail 138243
compact_success 6874
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 8443
unevictable_pgs_scanned 0
unevictable_pgs_rescued 7075596
unevictable_pgs_mlocked 7082893
unevictable_pgs_munlocked 7076462
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0
thp_fault_alloc 15563
thp_fault_fallback 59524
thp_collapse_alloc 1356
thp_collapse_alloc_failed 4821
thp_split 513

# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       2025976 146416  100
/dev/zram1                              partition       2025976 146108  100

# cat /sys/block/zram*/compr_data_size
47152674
47090766

-- 
Hubert Kario

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]