Anyone? Is this not the correct mailing list for this problem? If yes, are there any tools to help diagnose it? Where should I begin? I can reproduce this problem regularly, but it takes quite a bit of time, so I don't expect other people to diagnose and fix it. Regards, Hubert On Tuesday 22 of January 2013 02:50:40 Hubert Kario wrote: > Hi! > > In short, how to get detailed information what is the "noncache" column in > "kernel dynamic memory" in smem -w output? > > # smem -wtk > Area Used Cache Noncache > firmware/hardware 0 0 0 > kernel image 0 0 0 > kernel dynamic memory 1.5G 637.7M 853.5M > userspace memory 2.0G 142.7M 1.9G > free memory 383.5M 383.5M 0 > ---------------------------------------------------------- > 3.9G 1.1G 2.7G > > Little background info: > > I'm experiencing a weird problem with the kernel, where it does allocate > lots of memory, but I can't find what it is used for. > After the system has been running for few days (around a week usually) it > starts to swap quite heavily (if I don't close the biggest applications > quickly to recover I usually have to reset it as even remote ssh access > doesn't work, it responds to pings so it's not a hard lock up, Magic SysRq > doesn't restart it though). Thing is, it's enough to log-out from the X > environment and back in to make it go back to normal (where the noncache > memory is less than 200MiB). > > It's a workstation, with KMail, Opera and the usual background KDE > services running in the back (with file indexing turned off). Any other > processes are either short lived or not using any big amounts (>5MiB) of > memory. Archlinux, so the packages (both kernel and userspace) are either > vanilla or with minimal patches. > > The sum of USS of all processes in the system, as reported by smem > (the sum of RSS from `ps` is only 10-15% larger), during all that time > oscillates around 2GiB and basically never reaches 3GiB. I have tried > running with zram and it made the situation a bit better, but it > sometimes still locks up (because it starts swapping so hard). > > As restarting X helps, it led me to believe it may be a regression from > this bug: https://bugzilla.redhat.com/show_bug.cgi?id=615505 > but it doesn't look like it to me: > > # cat /sys/kernel/debug/dri/0/gem_names > name size handles refcount > 1 7680000 1 3 > 2 15564800 1 3 > > Some statistics and information on the system follow. > > Regards, > Hubert Kario > > # uname -a > Linux bursa22 3.6.11-1-ARCH #1 SMP PREEMPT Tue Dec 18 08:57:15 CET 2012 > x86_64 GNU/Linux > > # free -k > total used free shared buffers cached > Mem: 4051956 3598176 453780 0 30264 646828 > -/+ buffers/cache: 2921084 1130872 > Swap: 4051952 292524 3759428 > > # lsmod > Module Size Used by > vboxdrv 1823470 0 > iptable_filter 1457 1 > iptable_mangle 1585 1 > xt_tcpudp 2472 14 > xt_connmark 1910 12 > xt_mark 1214 13 > iptable_nat 4025 1 > nf_nat 15371 1 iptable_nat > nf_conntrack_ipv4 7799 15 nf_nat,iptable_nat > nf_defrag_ipv4 1340 1 nf_conntrack_ipv4 > nf_conntrack 64101 4 > nf_nat,xt_connmark,iptable_nat,nf_conntrack_ipv4 ip_tables > 16947 3 iptable_filter,iptable_mangle,iptable_nat x_tables > 17000 7 > xt_mark,ip_tables,xt_tcpudp,iptable_filter,xt_connmark,iptable_mangle,ipt > able_nat tun 16593 2 > zram 9460 2 > xfs 748208 1 > snd_hda_codec_hdmi 24529 1 > fuse 69213 5 > btrfs 718583 2 > coretemp 6071 0 > kvm_intel 124718 0 > crc32c 1769 1 > libcrc32c 1003 1 btrfs > zlib_deflate 20437 1 btrfs > kvm 374014 1 kvm_intel > radeon 859687 2 > iTCO_wdt 5256 0 > iTCO_vendor_support 1930 1 iTCO_wdt > gpio_ich 4064 0 > snd_hda_codec_realtek 61420 1 > ttm 64385 1 radeon > drm_kms_helper 32878 1 radeon > drm 218903 4 ttm,drm_kms_helper,radeon > r8169 56872 0 > i2c_algo_bit 5392 1 radeon > snd_hda_intel 26181 4 > mii 4092 1 r8169 > snd_hda_codec 98034 3 > snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_intel snd_hwdep > 6429 1 snd_hda_codec > snd_pcm 75735 3 > snd_hda_codec_hdmi,snd_hda_codec,snd_hda_intel snd_page_alloc > 7218 2 snd_pcm,snd_hda_intel > snd_timer 18935 1 snd_pcm > acpi_cpufreq 5934 0 > snd 60189 15 > snd_hda_codec_realtek,snd_hwdep,snd_timer,snd_hda_codec_hdmi,snd_pcm,snd_ > hda_codec,snd_hda_intel mperf 1300 1 acpi_cpufreq > intel_agp 10745 0 > i2c_i801 9572 0 > soundcore 5443 1 snd > intel_gtt 15660 1 intel_agp > microcode 12346 0 > i2c_core 20708 5 > drm,i2c_i801,drm_kms_helper,i2c_algo_bit,radeon lpc_ich > 10610 0 > pcspkr 1900 0 > processor 26856 3 acpi_cpufreq > button 4663 0 > evdev 10267 7 > ext4 440435 4 > crc16 1360 1 ext4 > jbd2 78802 1 ext4 > mbcache 6027 1 ext4 > dm_mod 72106 25 > raid456 55308 1 > async_raid6_recov 5375 1 raid456 > async_memcpy 1367 2 raid456,async_raid6_recov > async_pq 3261 2 raid456,async_raid6_recov > async_xor 2394 3 async_pq,raid456,async_raid6_recov > xor 8866 1 async_xor > async_tx 1903 5 > async_pq,raid456,async_xor,async_memcpy,async_raid6_recov raid6_pq > 86983 2 async_pq,async_raid6_recov > raid10 40365 1 > md_mod 104017 3 raid456,raid10 > hid_generic 1114 0 > usbhid 37036 0 > hid 85974 2 hid_generic,usbhid > sr_mod 14824 0 > cdrom 35521 1 sr_mod > sd_mod 29560 27 > ata_generic 3339 0 > uhci_hcd 23437 0 > pata_acpi 3452 0 > sata_via 8124 5 > ahci 21361 12 > libahci 20024 1 ahci > pata_it8213 3587 1 > firewire_ohci 31942 0 > libata 167757 6 > ahci,pata_acpi,sata_via,libahci,ata_generic,pata_it8213 scsi_mod > 133434 3 libata,sd_mod,sr_mod > firewire_core 53174 1 firewire_ohci > crc_itu_t 1364 1 firewire_core > ehci_hcd 41849 0 > usbcore 150472 3 uhci_hcd,ehci_hcd,usbhid > usb_common 955 1 usbcore > > # smem -kt | sed -n '1p;$p' > PID User Command Swap USS PSS > RSS 110 6 281.6M 2.0G 2.0G > 2.3G > > # cat /proc/meminfo > MemTotal: 4051956 kB > MemFree: 452512 kB > Buffers: 29784 kB > Cached: 648796 kB > SwapCached: 144308 kB > Active: 1555176 kB > Inactive: 1175416 kB > Active(anon): 1238968 kB > Inactive(anon): 868836 kB > Active(file): 316208 kB > Inactive(file): 306580 kB > Unevictable: 7720 kB > Mlocked: 7720 kB > SwapTotal: 4051952 kB > SwapFree: 3759416 kB > Dirty: 372 kB > Writeback: 0 kB > AnonPages: 1982576 kB > Mapped: 148724 kB > Shmem: 52248 kB > Slab: 185580 kB > SReclaimable: 69316 kB > SUnreclaim: 116264 kB > KernelStack: 3320 kB > PageTables: 43996 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 6077928 kB > Committed_AS: 3751688 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 139780 kB > VmallocChunk: 34359570976 kB > HardwareCorrupted: 0 kB > AnonHugePages: 442368 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 2536128 kB > DirectMap2M: 1656832 kB > > # cat /proc/vmstat > nr_free_pages 113256 > nr_inactive_anon 217210 > nr_active_anon 309694 > nr_inactive_file 76546 > nr_active_file 79049 > nr_unevictable 1930 > nr_mlock 1930 > nr_anon_pages 385048 > nr_mapped 37179 > nr_file_pages 205623 > nr_dirty 24 > nr_writeback 0 > nr_slab_reclaimable 17329 > nr_slab_unreclaimable 29066 > nr_page_table_pages 10973 > nr_kernel_stack 413 > nr_unstable 0 > nr_bounce 0 > nr_vmscan_write 3103053 > nr_vmscan_immediate_reclaim 21226934 > nr_writeback_temp 0 > nr_isolated_anon 0 > nr_isolated_file 0 > nr_shmem 13062 > nr_dirtied 39768555 > nr_written 38668348 > numa_hit 1043570022 > numa_miss 0 > numa_foreign 0 > numa_interleave 7629 > numa_local 1043570022 > numa_other 0 > nr_anon_transparent_hugepages 216 > nr_dirty_threshold 76584 > nr_dirty_background_threshold 38292 > pgpgin 538657436 > pgpgout 172521740 > pswpin 350320 > pswpout 404062 > pgalloc_dma 2 > pgalloc_dma32 666912295 > pgalloc_normal 571432892 > pgalloc_movable 0 > pgfree 1243823469 > pgactivate 73410706 > pgdeactivate 72896276 > pgfault 544723643 > pgmajfault 2023827 > pgrefill_dma 0 > pgrefill_dma32 64507672 > pgrefill_normal 18650343 > pgrefill_movable 0 > pgsteal_kswapd_dma 0 > pgsteal_kswapd_dma32 3005794 > pgsteal_kswapd_normal 26891312 > pgsteal_kswapd_movable 0 > pgsteal_direct_dma 0 > pgsteal_direct_dma32 15653023 > pgsteal_direct_normal 42635574 > pgsteal_direct_movable 0 > pgscan_kswapd_dma 0 > pgscan_kswapd_dma32 3136881 > pgscan_kswapd_normal 32518914 > pgscan_kswapd_movable 0 > pgscan_direct_dma 0 > pgscan_direct_dma32 361983838 > pgscan_direct_normal 6556538987 > pgscan_direct_movable 0 > pgscan_direct_throttle 0 > zone_reclaim_failed 0 > pginodesteal 24512 > slabs_scanned 35039616 > kswapd_inodesteal 193648 > kswapd_low_wmark_hit_quickly 654504 > kswapd_high_wmark_hit_quickly 690974 > kswapd_skip_congestion_wait 2977 > pageoutrun 645618572 > allocstall 7647697 > pgrotated 285215 > compact_blocks_moved 23846881 > compact_pages_moved 4897380 > compact_pagemigrate_failed 48162348 > compact_stall 145117 > compact_fail 138243 > compact_success 6874 > htlb_buddy_alloc_success 0 > htlb_buddy_alloc_fail 0 > unevictable_pgs_culled 8443 > unevictable_pgs_scanned 0 > unevictable_pgs_rescued 7075596 > unevictable_pgs_mlocked 7082893 > unevictable_pgs_munlocked 7076462 > unevictable_pgs_cleared 0 > unevictable_pgs_stranded 0 > unevictable_pgs_mlockfreed 0 > thp_fault_alloc 15563 > thp_fault_fallback 59524 > thp_collapse_alloc 1356 > thp_collapse_alloc_failed 4821 > thp_split 513 > > # cat /proc/swaps > Filename Type Size Used > Priority /dev/zram0 partition 2025976 > 146416 100 /dev/zram1 partition > 2025976 146108 100 > > # cat /sys/block/zram*/compr_data_size > 47152674 > 47090766 -- Hubert Kario -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>