Hi, We are running Linux 3.10 + fscache-20130702 + commit bae6b235905ab9dc6659395f7802c1d36fb63f15 from dhowells' git tree and suspect there might be a page reference leak somewhere as evidenced by the low reclaim ratio. We also see the kernel complaining about memory allocation failures in the readahead code as seen below. The machine has plenty of filecache and there isn't any heavy write activity which could also pin the pages. We wrote a script to monitor /proc/vmstat and see the following output with a one second interval. Refill Scan Free FrRatio Slab Runs Stalls 0 154663 6855 4 1024 2 0 0 7152 7152 100 0 4 0 0 5960 5960 100 0 4 0 0 7152 7152 100 1024 3 0 0 152407 21698 14 1024 2 0 The fields are: Refill - # active pages scanned, i.e. deactivated Scan - # inactive pages scanned Free - # pages freed FrRatio - free / scan ratio as percentage Runs - # times kswapd ran Stalls - # times direct reclaim ran The free memory looks like this (free -m): total used free shared buffers cached Mem: 24175 23986 188 0 16 10453 -/+ buffers/cache: 13517 10657 Swap: 8191 85 8106 Are you aware of any such issue with fscache / cachefiles? If not, could you suggest what other information we could gather to debug it further? Thanks, Shantanu SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z) python: page allocation failure: order:0, mode:0x11110 CPU: 5 PID: 18997 Comm: python Tainted: G W O 3.10.36-el5.ia32e.lime.0 #1 Hardware name: Supermicro X8STi/X8STi, BIOS 2.0 06/03/2010 0000000000000000 ffff880104d39578 ffffffff81426c43 ffff880104d39608 ffffffff810fdcfc ffff88061fffd468 0000004000000000 ffff880104d395a8 ffffffff810fd506 0000000000000000 ffffffff8109120d 0001111000000000 Call Trace: [<ffffffff81426c43>] dump_stack+0x19/0x1e [<ffffffff810fdcfc>] warn_alloc_failed+0xfc/0x140 [<ffffffff810fd506>] ? drain_local_pages+0x16/0x20 [<ffffffff8109120d>] ? on_each_cpu_mask+0x4d/0x70 [<ffffffff810fef1c>] __alloc_pages_nodemask+0x6cc/0x910 [<ffffffff81139a9a>] alloc_pages_current+0xba/0x160 [<ffffffff810f6617>] __page_cache_alloc+0xa7/0xc0 [<ffffffffa050a4ea>] cachefiles_read_backing_file+0x2ba/0x7b0 [cachefiles] [<ffffffffa050ac87>] cachefiles_read_or_alloc_pages+0x2a7/0x3d0 [cachefiles] [<ffffffff81062c4f>] ? wake_up_bit+0x2f/0x40 [<ffffffffa04da24a>] ? fscache_run_op+0x5a/0xa0 [fscache] [<ffffffffa04daacb>] ? fscache_submit_op+0x1db/0x4c0 [fscache] [<ffffffffa04dbd65>] __fscache_read_or_alloc_pages+0x1f5/0x2c0 [fscache] [<ffffffffa0531e9e>] __nfs_readpages_from_fscache+0x7e/0x1b0 [nfs] [<ffffffffa052bc6a>] nfs_readpages+0xca/0x1f0 [nfs] [<ffffffff81139a9a>] ? alloc_pages_current+0xba/0x160 [<ffffffff81101ae2>] __do_page_cache_readahead+0x1b2/0x260 [<ffffffff81101bb1>] ra_submit+0x21/0x30 [<ffffffff81101e85>] ondemand_readahead+0x115/0x240 [<ffffffff81102038>] page_cache_async_readahead+0x88/0xb0 [<ffffffff810f5f5e>] ? find_get_page+0x1e/0xa0 [<ffffffff810f81fc>] generic_file_aio_read+0x4dc/0x720 [<ffffffffa05224b9>] nfs_file_read+0x89/0x100 [nfs] [<ffffffff81158a4f>] do_sync_read+0x7f/0xb0 [<ffffffff8115a3a5>] vfs_read+0xc5/0x190 [<ffffffff8115a57f>] SyS_read+0x5f/0xa0 [<ffffffff814327bb>] tracesys+0xdd/0xe2 Mem-Info: Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 2 CPU 1: hi: 186, btch: 31 usd: 159 CPU 2: hi: 186, btch: 31 usd: 82 CPU 3: hi: 186, btch: 31 usd: 182 CPU 4: hi: 186, btch: 31 usd: 57 CPU 5: hi: 186, btch: 31 usd: 0 Node 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 22 CPU 1: hi: 186, btch: 31 usd: 117 CPU 2: hi: 186, btch: 31 usd: 16 CPU 3: hi: 186, btch: 31 usd: 79 CPU 4: hi: 186, btch: 31 usd: 59 CPU 5: hi: 186, btch: 31 usd: 0 active_anon:3461555 inactive_anon:451550 isolated_anon:0 active_file:236104 inactive_file:1899298 isolated_file:0 unevictable:467 dirty:109 writeback:0 unstable:0 free:33860 slab_reclaimable:19654 slab_unreclaimable:5110 mapped:15284 shmem:2 pagetables:10231 bounce:0 free_cma:0 Node 0 DMA32 free:91764kB min:4824kB low:6028kB high:7236kB active_anon:2185528kB inactive_anon:566976kB active_file:108808kB inactive_file:670996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3660960kB managed:3646160kB mlocked:0kB dirty:152kB writeback:0kB mapped:1344kB shmem:0kB slab_reclaimable:17300kB slab_unreclaimable:580kB kernel_stack:80kB pagetables:6448kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 20607 20607 Node 0 Normal free:39164kB min:27940kB low:34924kB high:41908kB active_anon:11663332kB inactive_anon:1239224kB active_file:835608kB inactive_file:6926636kB unevictable:1868kB isolated(anon):0kB isolated(file):0kB present:21495808kB managed:21101668kB mlocked:1868kB dirty:612kB writeback:0kB mapped:60120kB shmem:8kB slab_reclaimable:61536kB slab_unreclaimable:19860kB kernel_stack:2088kB pagetables:34804kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 Node 0 DMA32: 506*4kB (UEM) 545*8kB (UEM) 569*16kB (UEM) 897*32kB (UEM) 541*64kB (UEM) 87*128kB (UM) 4*256kB (UM) 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 92000kB Node 0 Normal: 703*4kB (UE) 285*8kB (UEM) 176*16kB (UEM) 836*32kB (UEM) 7*64kB (UEM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35108kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 2137331 total pagecache pages 1215 pages in swap cache Swap cache stats: add 7056879, delete 7055664, find 633567/703482 Free swap = 8301172kB Total swap = 8388604kB 6291455 pages RAM 102580 pages reserved 4391935 pages shared 4746779 pages non-shared -- Linux-cachefs mailing list Linux-cachefs@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cachefs
