message log: Sep 25 11:37:30 ct2 kernel: rbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 Sep 25 11:37:30 ct2 kernel: rbd cpuset=/ mems_allowed=0-1 Sep 25 11:37:30 ct2 kernel: Pid: 28217, comm: rbd Not tainted 2.6.32-431.el6.x86_64 #1 Sep 25 11:37:30 ct2 kernel: Call Trace: Sep 25 11:37:30 ct2 kernel: [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122cde>] ? select_bad_process+0x9e/0x120 Sep 25 11:37:30 ct2 kernel: [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81167b9a>] ? alloc_pages_vma+0x9a/0x150 Sep 25 11:37:30 ct2 kernel: [<ffffffff8114ac9d>] ? handle_pte_fault+0x73d/0xb00 Sep 25 11:37:30 ct2 kernel: [<ffffffff8104eeb7>] ? pte_alloc_one+0x37/0x50 Sep 25 11:37:30 ct2 kernel: [<ffffffff811832e9>] ? do_huge_pmd_anonymous_page+0xb9/0x3b0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8114b28a>] ? handle_mm_fault+0x22a/0x300 Sep 25 11:37:30 ct2 kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480 Sep 25 11:37:30 ct2 kernel: [<ffffffff81151825>] ? do_mmap_pgoff+0x335/0x380 Sep 25 11:37:30 ct2 kernel: [<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8152a815>] ? page_fault+0x25/0x30 Sep 25 11:37:30 ct2 kernel: Mem-Info: Sep 25 11:37:30 ct2 kernel: Node 0 DMA per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 0 Normal per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 1 Normal per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: active_anon:11237039 inactive_anon:870686 isolated_anon:0 Sep 25 11:37:30 ct2 kernel: active_file:981 inactive_file:1095 isolated_file:0 Sep 25 11:37:30 ct2 kernel: unevictable:0 dirty:68 writeback:971 unstable:0 Sep 25 11:37:30 ct2 kernel: free:47328 slab_reclaimable:10413 slab_unreclaimable:35885 Sep 25 11:37:30 ct2 kernel: mapped:1017 shmem:1 pagetables:39376 bounce:0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA free:15740kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 3211 24168 24168 Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 free:89632kB min:5972kB low:7464kB high:8956kB active_anon:2137168kB inactive_anon:534280kB active_file:12kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3288224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:2204kB slab_unreclaimable:9540kB kernel_stack:176kB pagetables:5044kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 20957 20957 Sep 25 11:37:30 ct2 kernel: Node 0 Normal free:38944kB min:38996kB low:48744kB high:58492kB active_anon:19871956kB inactive_anon:1419396kB active_file:4kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21460480kB mlocked:0kB dirty:8kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:19504kB slab_unreclaimable:81520kB kernel_stack:15768kB pagetables:108956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 25 11:37:30 ct2 kernel: Node 1 Normal free:44996kB min:45108kB low:56384kB high:67660kB active_anon:22939032kB inactive_anon:1529068kB active_file:3908kB inactive_file:4280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:24821760kB mlocked:0kB dirty:264kB writeback:3884kB mapped:4064kB shmem:4kB slab_reclaimable:19944kB slab_unreclaimable:52480kB kernel_stack:7968kB pagetables:43504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4288 all_unreclaimable? no Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB Sep 25 11:37:30 ct2 kernel: Node 0 DMA32: 732*4kB 600*8kB 474*16kB 385*32kB 261*64kB 159*128kB 67*256kB 16*512kB 0*1024kB 0*2048kB 0*4096kB = 90032kB Sep 25 11:37:30 ct2 kernel: Node 0 Normal: 9788*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 39152kB Sep 25 11:37:30 ct2 kernel: Node 1 Normal: 11291*4kB 55*8kB 19*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45940kB Sep 25 11:37:30 ct2 kernel: 10737 total pagecache pages Sep 25 11:37:30 ct2 kernel: 8671 pages in swap cache Sep 25 11:37:30 ct2 kernel: Swap cache stats: add 6400009, delete 6391338, find 89617/105665 Sep 25 11:37:30 ct2 kernel: Free swap = 0kB Sep 25 11:37:30 ct2 kernel: Total swap = 24756216kB Sep 25 11:37:30 ct2 kernel: 12582911 pages RAM Sep 25 11:37:30 ct2 kernel: 239403 pages reserved Sep 25 11:37:30 ct2 kernel: 9453 pages shared Sep 25 11:37:30 ct2 kernel: 12280860 pages non-shared Sep 25 11:37:30 ct2 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Sep 25 11:37:30 ct2 kernel: [ 727] 0 727 2791 0 2 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [ 1590] 0 1590 63856 38 1 0 0 rsyslogd Sep 25 11:37:30 ct2 kernel: [ 1737] 0 1737 20318 24 6 0 0 master Sep 25 11:37:30 ct2 kernel: [ 1744] 89 1744 20381 19 0 0 0 qmgr Sep 25 11:37:30 ct2 kernel: [ 1747] 0 1747 29333 24 0 0 0 crond Sep 25 11:37:30 ct2 kernel: [ 1760] 0 1760 14211 1 0 0 0 login Sep 25 11:37:30 ct2 kernel: [ 1762] 0 1762 1016 1 5 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1764] 0 1764 1016 1 3 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1766] 0 1766 1016 1 0 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1768] 0 1768 1016 1 6 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1770] 0 1770 1016 1 4 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1784] 0 1784 27076 25 5 0 0 bash Sep 25 11:37:30 ct2 kernel: [ 4915] 0 4915 2280 1 4 0 0 dhclient Sep 25 11:37:30 ct2 kernel: [ 4958] 0 4958 25089 37 2 0 0 sshd Sep 25 11:37:30 ct2 kernel: [ 4962] 0 4962 27101 12 3 0 0 bash Sep 25 11:37:30 ct2 kernel: [ 5025] 0 5025 16651 0 4 -17 -1000 sshd Sep 25 11:37:30 ct2 kernel: [ 5688] 0 5688 6910 10 1 -17 -1000 auditd Sep 25 11:37:30 ct2 kernel: [17539] 0 17539 25266 0 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17540] 0 17540 20144 0 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17542] 0 17542 23264 1 5 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17543] 0 17543 39609 1 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17932] 0 17932 39609 1 3 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17937] 0 17937 39603 1 2 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17940] 0 17940 40123 1 0 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [18632] 0 18632 291742 269 2 0 0 dsm_sa_datamgrd Sep 25 11:37:30 ct2 kernel: [18780] 0 18780 73312 86 4 0 0 dsm_sa_eventmgr Sep 25 11:37:30 ct2 kernel: [18796] 0 18796 171833 21 0 0 0 dsm_sa_datamgrd Sep 25 11:37:30 ct2 kernel: [18842] 0 18842 109528 108 4 0 0 dsm_sa_snmpd Sep 25 11:37:30 ct2 kernel: [19147] 0 19147 33251 1 0 0 0 dsm_om_connsvcd Sep 25 11:37:30 ct2 kernel: [19148] 0 19148 831391 28062 1 0 0 dsm_om_connsvcd Sep 25 11:37:30 ct2 kernel: [38380] 0 38380 25091 4 4 0 0 sshd Sep 25 11:37:30 ct2 kernel: [38386] 0 38386 27076 4 1 0 0 bash Sep 25 11:37:30 ct2 kernel: [28907] 0 28907 2883 0 5 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [28908] 0 28908 2856 0 7 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [44087] 0 44087 24573 7 0 0 0 sshd Sep 25 11:37:30 ct2 kernel: [44090] 0 44090 27119 8 1 0 0 bash Sep 25 11:37:30 ct2 kernel: [34590] 38 34590 6624 34 2 0 0 ntpd Sep 25 11:37:30 ct2 kernel: [34920] 0 34920 84080 11471 3 0 0 ceph-mon Sep 25 11:37:30 ct2 kernel: [35674] 0 35674 268185 42473 1 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [43044] 0 43044 247471 39531 5 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [ 7125] 0 7125 270942 41559 1 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [20854] 0 20854 251666 40220 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [21799] 0 21799 250641 29693 5 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [22800] 0 22800 275004 33343 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [23509] 0 23509 262113 28210 4 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [24445] 0 24445 259672 33212 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [25146] 0 25146 266437 35462 2 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [25889] 0 25889 257854 34321 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [27977] 0 27977 166624 482 2 0 0 python Sep 25 11:37:30 ct2 kernel: [28216] 0 28216 1918 12 6 0 0 nc Sep 25 11:37:30 ct2 kernel: [28217] 0 28217 18681959 11707670 2 0 0 rbd Sep 25 11:37:30 ct2 kernel: [28636] 89 28636 20344 127 4 0 0 pickup Sep 25 11:37:30 ct2 kernel: Out of memory: Kill process 28217 (rbd) score 920 or sacrifice child Sep 25 11:37:30 ct2 kernel: Killed process 28217, UID 0, (rbd) total-vm:74727836kB, anon-rss:46829404kB, file-rss:1276kB Sep 25 11:37:30 ct2 kernel: ceph-osd invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0 Sep 25 11:37:30 ct2 kernel: ceph-osd cpuset=/ mems_allowed=0-1 Sep 25 11:37:30 ct2 kernel: Pid: 5875, comm: ceph-osd Not tainted 2.6.32-431.el6.x86_64 #1 Sep 25 11:37:30 ct2 kernel: Call Trace: Sep 25 11:37:30 ct2 kernel: [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81122cde>] ? select_bad_process+0x9e/0x120 Sep 25 11:37:30 ct2 kernel: [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0 Sep 25 11:37:30 ct2 kernel: [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0 Sep 25 11:37:30 ct2 kernel: [<ffffffff81167a9a>] ? alloc_pages_current+0xaa/0x110 Sep 25 11:37:30 ct2 kernel: [<ffffffff814a2297>] ? tcp_sendmsg+0x677/0xa20 Sep 25 11:37:30 ct2 kernel: [<ffffffff81447e03>] ? sock_sendmsg+0x123/0x150 Sep 25 11:37:30 ct2 kernel: [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40 Sep 25 11:37:30 ct2 kernel: [<ffffffff814495f6>] ? __sys_sendmsg+0x406/0x420 Sep 25 11:37:30 ct2 kernel: [<ffffffff8104a98c>] ? __do_page_fault+0x1ec/0x480 Sep 25 11:37:30 ct2 kernel: [<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13 Sep 25 11:37:30 ct2 kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20 Sep 25 11:37:30 ct2 kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20 Sep 25 11:37:30 ct2 kernel: [<ffffffff81449819>] ? sys_sendmsg+0x49/0x90 Sep 25 11:37:30 ct2 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b Sep 25 11:37:30 ct2 kernel: Mem-Info: Sep 25 11:37:30 ct2 kernel: Node 0 DMA per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 0, btch: 1 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 53 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 30 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 0 Normal per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 21 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 30 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: Node 1 Normal per-cpu: Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 10 Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 Sep 25 11:37:30 ct2 kernel: active_anon:11237039 inactive_anon:870686 isolated_anon:0 Sep 25 11:37:30 ct2 kernel: active_file:981 inactive_file:1095 isolated_file:0 Sep 25 11:37:30 ct2 kernel: unevictable:0 dirty:68 writeback:753 unstable:0 Sep 25 11:37:30 ct2 kernel: free:47314 slab_reclaimable:10413 slab_unreclaimable:35885 Sep 25 11:37:30 ct2 kernel: mapped:1017 shmem:1 pagetables:39376 bounce:0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA free:15740kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 3211 24168 24168 Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 free:89576kB min:5972kB low:7464kB high:8956kB active_anon:2137168kB inactive_anon:534280kB active_file:12kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3288224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:2204kB slab_unreclaimable:9540kB kernel_stack:176kB pagetables:5044kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 20957 20957 Sep 25 11:37:30 ct2 kernel: Node 0 Normal free:38944kB min:38996kB low:48744kB high:58492kB active_anon:19871956kB inactive_anon:1419396kB active_file:4kB inactive_file:100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21460480kB mlocked:0kB dirty:8kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:19504kB slab_unreclaimable:81520kB kernel_stack:15768kB pagetables:108956kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 25 11:37:30 ct2 kernel: Node 1 Normal free:44996kB min:45108kB low:56384kB high:67660kB active_anon:22939032kB inactive_anon:1529068kB active_file:3908kB inactive_file:4280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:24821760kB mlocked:0kB dirty:264kB writeback:3012kB mapped:4064kB shmem:4kB slab_reclaimable:19944kB slab_unreclaimable:52480kB kernel_stack:7968kB pagetables:43504kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12544 all_unreclaimable? yes Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 Sep 25 11:37:30 ct2 kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB Sep 25 11:37:30 ct2 kernel: Node 0 DMA32: 638*4kB 601*8kB 469*16kB 384*32kB 260*64kB 159*128kB 67*256kB 16*512kB 0*1024kB 0*2048kB 0*4096kB = 89488kB Sep 25 11:37:30 ct2 kernel: Node 0 Normal: 9695*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 38780kB Sep 25 11:37:30 ct2 kernel: Node 1 Normal: 11291*4kB 55*8kB 19*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45940kB Sep 25 11:37:30 ct2 kernel: 10737 total pagecache pages Sep 25 11:37:30 ct2 kernel: 8679 pages in swap cache Sep 25 11:37:30 ct2 kernel: Swap cache stats: add 6400017, delete 6391338, find 89617/105666 Sep 25 11:37:30 ct2 kernel: Free swap = 0kB Sep 25 11:37:30 ct2 kernel: Total swap = 24756216kB Sep 25 11:37:30 ct2 kernel: 12582911 pages RAM Sep 25 11:37:30 ct2 kernel: 239403 pages reserved Sep 25 11:37:30 ct2 kernel: 9358 pages shared Sep 25 11:37:30 ct2 kernel: 12281029 pages non-shared Sep 25 11:37:30 ct2 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Sep 25 11:37:30 ct2 kernel: [ 727] 0 727 2791 0 2 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [ 1590] 0 1590 63856 38 1 0 0 rsyslogd Sep 25 11:37:30 ct2 kernel: [ 1737] 0 1737 20318 24 6 0 0 master Sep 25 11:37:30 ct2 kernel: [ 1744] 89 1744 20381 19 0 0 0 qmgr Sep 25 11:37:30 ct2 kernel: [ 1747] 0 1747 29333 24 0 0 0 crond Sep 25 11:37:30 ct2 kernel: [ 1760] 0 1760 14211 1 0 0 0 login Sep 25 11:37:30 ct2 kernel: [ 1762] 0 1762 1016 1 5 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1764] 0 1764 1016 1 3 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1766] 0 1766 1016 1 0 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1768] 0 1768 1016 1 6 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1770] 0 1770 1016 1 4 0 0 mingetty Sep 25 11:37:30 ct2 kernel: [ 1784] 0 1784 27076 25 5 0 0 bash Sep 25 11:37:30 ct2 kernel: [ 4915] 0 4915 2280 1 4 0 0 dhclient Sep 25 11:37:30 ct2 kernel: [ 4958] 0 4958 25089 37 2 0 0 sshd Sep 25 11:37:30 ct2 kernel: [ 4962] 0 4962 27101 12 3 0 0 bash Sep 25 11:37:30 ct2 kernel: [ 5025] 0 5025 16651 0 4 -17 -1000 sshd Sep 25 11:37:30 ct2 kernel: [ 5688] 0 5688 6910 10 1 -17 -1000 auditd Sep 25 11:37:30 ct2 kernel: [17539] 0 17539 25266 0 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17540] 0 17540 20144 0 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17542] 0 17542 23264 1 5 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17543] 0 17543 39609 1 1 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17932] 0 17932 39609 1 3 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17937] 0 17937 39603 1 2 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [17940] 0 17940 40123 1 0 0 0 sfcbd Sep 25 11:37:30 ct2 kernel: [18632] 0 18632 291742 269 2 0 0 dsm_sa_datamgrd Sep 25 11:37:30 ct2 kernel: [18780] 0 18780 73312 86 4 0 0 dsm_sa_eventmgr Sep 25 11:37:30 ct2 kernel: [18796] 0 18796 171833 21 0 0 0 dsm_sa_datamgrd Sep 25 11:37:30 ct2 kernel: [18842] 0 18842 109528 108 4 0 0 dsm_sa_snmpd Sep 25 11:37:30 ct2 kernel: [19147] 0 19147 33251 1 0 0 0 dsm_om_connsvcd Sep 25 11:37:30 ct2 kernel: [19148] 0 19148 831391 28062 1 0 0 dsm_om_connsvcd Sep 25 11:37:30 ct2 kernel: [38380] 0 38380 25091 4 4 0 0 sshd Sep 25 11:37:30 ct2 kernel: [38386] 0 38386 27076 4 1 0 0 bash Sep 25 11:37:30 ct2 kernel: [28907] 0 28907 2883 0 5 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [28908] 0 28908 2856 0 7 -17 -1000 udevd Sep 25 11:37:30 ct2 kernel: [44087] 0 44087 24573 7 0 0 0 sshd Sep 25 11:37:30 ct2 kernel: [44090] 0 44090 27119 8 1 0 0 bash Sep 25 11:37:30 ct2 kernel: [34590] 38 34590 6624 34 2 0 0 ntpd Sep 25 11:37:30 ct2 kernel: [34920] 0 34920 84080 11471 3 0 0 ceph-mon Sep 25 11:37:30 ct2 kernel: [35674] 0 35674 268185 42473 1 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [43044] 0 43044 247471 39531 5 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [ 7125] 0 7125 270942 41559 1 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [20854] 0 20854 251666 40220 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [21799] 0 21799 250641 29693 5 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [22800] 0 22800 275004 33343 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [23509] 0 23509 262113 28210 4 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [24445] 0 24445 259672 33212 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [25146] 0 25146 266437 35462 2 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [25889] 0 25889 257854 34321 0 0 0 ceph-osd Sep 25 11:37:30 ct2 kernel: [27977] 0 27977 166624 482 2 0 0 python Sep 25 11:37:30 ct2 kernel: [28216] 0 28216 1918 12 6 0 0 nc Sep 25 11:37:30 ct2 kernel: [28636] 89 28636 20344 127 4 0 0 pickup 2014-09-26 15:48 GMT+04:00 Irek Fasikhov <malmyzh at gmail.com>: > OS: CentOS 6.5 > Kernel: 2.6.32-431.el6.x86_64 > Ceph --version: ceph version 0.72.2 > (a913ded2ff138aefb8cb84d347d72164099cfd60) > > > > 2014-09-26 15:44 GMT+04:00 Irek Fasikhov <malmyzh at gmail.com>: > >> Hi, All. >> >> I see a memory leak when importing raw devi?e. >> >> Export Scheme: >> [rbd at rbdbackup ~]$ rbd --no-progress -n client.rbdbackup -k >> /etc/ceph/big.keyring -c /etc/ceph/big.conf export rbdtest/vm-111-disk-1 - >> | nc 10.43.255.252 12345 >> >> [root at ct2 ~]# nc -l 12345 | rbd import --no-progress --image-format 2 - >> rbd/vm-111-disk-1 >> >> This is the same problem with ssh >> >> Memory usage, see the screenshots: >> >> https://drive.google.com/folderview?id=0BxoNLVWxzOJWSHlTSEZvM3lkQXM&usp=sharing >> >> -- >> ? ?????????, ??????? ???? ??????????? >> ???.: +79229045757 >> > > > > -- > ? ?????????, ??????? ???? ??????????? > ???.: +79229045757 > -- ? ?????????, ??????? ???? ??????????? ???.: +79229045757 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140926/c0a3ad17/attachment.htm>
 |