I created a task in: http://tracker.ceph.com/issues/9602 2014-09-26 15:54 GMT+04:00 Irek Fasikhov <malmyzh at gmail.com>: > message log: > Sep 25 11:37:30 ct2 kernel: rbd invoked oom-killer: gfp_mask=0x280da, > order=0, oom_adj=0, oom_score_adj=0 > Sep 25 11:37:30 ct2 kernel: rbd cpuset=/ mems_allowed=0-1 > Sep 25 11:37:30 ct2 kernel: Pid: 28217, comm: rbd Not tainted > 2.6.32-431.el6.x86_64 #1 > Sep 25 11:37:30 ct2 kernel: Call Trace: > Sep 25 11:37:30 ct2 kernel: [<ffffffff810d05b1>] ? > cpuset_print_task_mems_allowed+0x91/0xb0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8122798c>] ? > security_real_capable_noaudit+0x3c/0x70 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122de2>] ? > oom_kill_process+0x82/0x2a0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122cde>] ? > select_bad_process+0x9e/0x120 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81123220>] ? > out_of_memory+0x220/0x3c0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8112fb3c>] ? > __alloc_pages_nodemask+0x8ac/0x8d0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81167b9a>] ? > alloc_pages_vma+0x9a/0x150 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8114ac9d>] ? > handle_pte_fault+0x73d/0xb00 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8104eeb7>] ? pte_alloc_one+0x37/0x50 > Sep 25 11:37:30 ct2 kernel: [<ffffffff811832e9>] ? > do_huge_pmd_anonymous_page+0xb9/0x3b0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8114b28a>] ? > handle_mm_fault+0x22a/0x300 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8104a8d8>] ? > __do_page_fault+0x138/0x480 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81151825>] ? > do_mmap_pgoff+0x335/0x380 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8152a815>] ? page_fault+0x25/0x30 > Sep 25 11:37:30 ct2 kernel: Mem-Info: > Sep 25 11:37:30 ct2 kernel: Node 0 DMA per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 0 Normal per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 1 Normal per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: active_anon:11237039 inactive_anon:870686 > isolated_anon:0 > Sep 25 11:37:30 ct2 kernel: active_file:981 inactive_file:1095 > isolated_file:0 > Sep 25 11:37:30 ct2 kernel: unevictable:0 dirty:68 writeback:971 unstable:0 > Sep 25 11:37:30 ct2 kernel: free:47328 slab_reclaimable:10413 > slab_unreclaimable:35885 > Sep 25 11:37:30 ct2 kernel: mapped:1017 shmem:1 pagetables:39376 bounce:0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA free:15740kB min:24kB low:28kB > high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? yes > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 3211 24168 24168 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 free:89632kB min:5972kB > low:7464kB high:8956kB active_anon:2137168kB inactive_anon:534280kB > active_file:12kB inactive_file:0kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:3288224kB mlocked:0kB dirty:0kB writeback:0kB > mapped:0kB shmem:0kB slab_reclaimable:2204kB slab_unreclaimable:9540kB > kernel_stack:176kB pagetables:5044kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 20957 20957 > Sep 25 11:37:30 ct2 kernel: Node 0 Normal free:38944kB min:38996kB > low:48744kB high:58492kB active_anon:19871956kB inactive_anon:1419396kB > active_file:4kB inactive_file:100kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:21460480kB mlocked:0kB dirty:8kB writeback:0kB > mapped:4kB shmem:0kB slab_reclaimable:19504kB slab_unreclaimable:81520kB > kernel_stack:15768kB pagetables:108956kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 > Sep 25 11:37:30 ct2 kernel: Node 1 Normal free:44996kB min:45108kB > low:56384kB high:67660kB active_anon:22939032kB inactive_anon:1529068kB > active_file:3908kB inactive_file:4280kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:24821760kB mlocked:0kB dirty:264kB > writeback:3884kB mapped:4064kB shmem:4kB slab_reclaimable:19944kB > slab_unreclaimable:52480kB kernel_stack:7968kB pagetables:43504kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4288 > all_unreclaimable? no > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB > 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32: 732*4kB 600*8kB 474*16kB > 385*32kB 261*64kB 159*128kB 67*256kB 16*512kB 0*1024kB 0*2048kB 0*4096kB = > 90032kB > Sep 25 11:37:30 ct2 kernel: Node 0 Normal: 9788*4kB 0*8kB 0*16kB 0*32kB > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 39152kB > Sep 25 11:37:30 ct2 kernel: Node 1 Normal: 11291*4kB 55*8kB 19*16kB 1*32kB > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45940kB > Sep 25 11:37:30 ct2 kernel: 10737 total pagecache pages > Sep 25 11:37:30 ct2 kernel: 8671 pages in swap cache > Sep 25 11:37:30 ct2 kernel: Swap cache stats: add 6400009, delete 6391338, > find 89617/105665 > Sep 25 11:37:30 ct2 kernel: Free swap = 0kB > Sep 25 11:37:30 ct2 kernel: Total swap = 24756216kB > Sep 25 11:37:30 ct2 kernel: 12582911 pages RAM > Sep 25 11:37:30 ct2 kernel: 239403 pages reserved > Sep 25 11:37:30 ct2 kernel: 9453 pages shared > Sep 25 11:37:30 ct2 kernel: 12280860 pages non-shared > Sep 25 11:37:30 ct2 kernel: [ pid ] uid tgid total_vm rss cpu > oom_adj oom_score_adj name > Sep 25 11:37:30 ct2 kernel: [ 727] 0 727 2791 0 2 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [ 1590] 0 1590 63856 38 1 > 0 0 rsyslogd > Sep 25 11:37:30 ct2 kernel: [ 1737] 0 1737 20318 24 6 > 0 0 master > Sep 25 11:37:30 ct2 kernel: [ 1744] 89 1744 20381 19 0 > 0 0 qmgr > Sep 25 11:37:30 ct2 kernel: [ 1747] 0 1747 29333 24 0 > 0 0 crond > Sep 25 11:37:30 ct2 kernel: [ 1760] 0 1760 14211 1 0 > 0 0 login > Sep 25 11:37:30 ct2 kernel: [ 1762] 0 1762 1016 1 5 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1764] 0 1764 1016 1 3 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1766] 0 1766 1016 1 0 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1768] 0 1768 1016 1 6 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1770] 0 1770 1016 1 4 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1784] 0 1784 27076 25 5 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [ 4915] 0 4915 2280 1 4 > 0 0 dhclient > Sep 25 11:37:30 ct2 kernel: [ 4958] 0 4958 25089 37 2 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [ 4962] 0 4962 27101 12 3 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [ 5025] 0 5025 16651 0 4 > -17 -1000 sshd > Sep 25 11:37:30 ct2 kernel: [ 5688] 0 5688 6910 10 1 > -17 -1000 auditd > Sep 25 11:37:30 ct2 kernel: [17539] 0 17539 25266 0 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17540] 0 17540 20144 0 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17542] 0 17542 23264 1 5 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17543] 0 17543 39609 1 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17932] 0 17932 39609 1 3 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17937] 0 17937 39603 1 2 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17940] 0 17940 40123 1 0 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [18632] 0 18632 291742 269 2 > 0 0 dsm_sa_datamgrd > Sep 25 11:37:30 ct2 kernel: [18780] 0 18780 73312 86 4 > 0 0 dsm_sa_eventmgr > Sep 25 11:37:30 ct2 kernel: [18796] 0 18796 171833 21 0 > 0 0 dsm_sa_datamgrd > Sep 25 11:37:30 ct2 kernel: [18842] 0 18842 109528 108 4 > 0 0 dsm_sa_snmpd > Sep 25 11:37:30 ct2 kernel: [19147] 0 19147 33251 1 0 > 0 0 dsm_om_connsvcd > Sep 25 11:37:30 ct2 kernel: [19148] 0 19148 831391 28062 1 > 0 0 dsm_om_connsvcd > Sep 25 11:37:30 ct2 kernel: [38380] 0 38380 25091 4 4 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [38386] 0 38386 27076 4 1 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [28907] 0 28907 2883 0 5 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [28908] 0 28908 2856 0 7 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [44087] 0 44087 24573 7 0 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [44090] 0 44090 27119 8 1 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [34590] 38 34590 6624 34 2 > 0 0 ntpd > Sep 25 11:37:30 ct2 kernel: [34920] 0 34920 84080 11471 3 > 0 0 ceph-mon > Sep 25 11:37:30 ct2 kernel: [35674] 0 35674 268185 42473 1 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [43044] 0 43044 247471 39531 5 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [ 7125] 0 7125 270942 41559 1 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [20854] 0 20854 251666 40220 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [21799] 0 21799 250641 29693 5 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [22800] 0 22800 275004 33343 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [23509] 0 23509 262113 28210 4 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [24445] 0 24445 259672 33212 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [25146] 0 25146 266437 35462 2 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [25889] 0 25889 257854 34321 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [27977] 0 27977 166624 482 2 > 0 0 python > Sep 25 11:37:30 ct2 kernel: [28216] 0 28216 1918 12 6 > 0 0 nc > Sep 25 11:37:30 ct2 kernel: [28217] 0 28217 18681959 11707670 2 > 0 0 rbd > Sep 25 11:37:30 ct2 kernel: [28636] 89 28636 20344 127 4 > 0 0 pickup > Sep 25 11:37:30 ct2 kernel: Out of memory: Kill process 28217 (rbd) score > 920 or sacrifice child > Sep 25 11:37:30 ct2 kernel: Killed process 28217, UID 0, (rbd) > total-vm:74727836kB, anon-rss:46829404kB, file-rss:1276kB > Sep 25 11:37:30 ct2 kernel: ceph-osd invoked oom-killer: gfp_mask=0xd0, > order=0, oom_adj=0, oom_score_adj=0 > Sep 25 11:37:30 ct2 kernel: ceph-osd cpuset=/ mems_allowed=0-1 > Sep 25 11:37:30 ct2 kernel: Pid: 5875, comm: ceph-osd Not tainted > 2.6.32-431.el6.x86_64 #1 > Sep 25 11:37:30 ct2 kernel: Call Trace: > Sep 25 11:37:30 ct2 kernel: [<ffffffff810d05b1>] ? > cpuset_print_task_mems_allowed+0x91/0xb0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8122798c>] ? > security_real_capable_noaudit+0x3c/0x70 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122de2>] ? > oom_kill_process+0x82/0x2a0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81122cde>] ? > select_bad_process+0x9e/0x120 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81123220>] ? > out_of_memory+0x220/0x3c0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8112fb3c>] ? > __alloc_pages_nodemask+0x8ac/0x8d0 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81167a9a>] ? > alloc_pages_current+0xaa/0x110 > Sep 25 11:37:30 ct2 kernel: [<ffffffff814a2297>] ? tcp_sendmsg+0x677/0xa20 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81447e03>] ? sock_sendmsg+0x123/0x150 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8109b2a0>] ? > autoremove_wake_function+0x0/0x40 > Sep 25 11:37:30 ct2 kernel: [<ffffffff814495f6>] ? > __sys_sendmsg+0x406/0x420 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8104a98c>] ? > __do_page_fault+0x1ec/0x480 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8100b9ce>] ? > common_interrupt+0xe/0x13 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8100bb8e>] ? > apic_timer_interrupt+0xe/0x20 > Sep 25 11:37:30 ct2 kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20 > Sep 25 11:37:30 ct2 kernel: [<ffffffff81449819>] ? sys_sendmsg+0x49/0x90 > Sep 25 11:37:30 ct2 kernel: [<ffffffff8100b072>] ? > system_call_fastpath+0x16/0x1b > Sep 25 11:37:30 ct2 kernel: Mem-Info: > Sep 25 11:37:30 ct2 kernel: Node 0 DMA per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 0, btch: 1 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 53 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 30 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 0 Normal per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 21 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 30 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: Node 1 Normal per-cpu: > Sep 25 11:37:30 ct2 kernel: CPU 0: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 1: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 2: hi: 186, btch: 31 usd: 10 > Sep 25 11:37:30 ct2 kernel: CPU 3: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 4: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 5: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 6: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: CPU 7: hi: 186, btch: 31 usd: 0 > Sep 25 11:37:30 ct2 kernel: active_anon:11237039 inactive_anon:870686 > isolated_anon:0 > Sep 25 11:37:30 ct2 kernel: active_file:981 inactive_file:1095 > isolated_file:0 > Sep 25 11:37:30 ct2 kernel: unevictable:0 dirty:68 writeback:753 unstable:0 > Sep 25 11:37:30 ct2 kernel: free:47314 slab_reclaimable:10413 > slab_unreclaimable:35885 > Sep 25 11:37:30 ct2 kernel: mapped:1017 shmem:1 pagetables:39376 bounce:0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA free:15740kB min:24kB low:28kB > high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB > inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB > present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB > slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 > all_unreclaimable? yes > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 3211 24168 24168 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32 free:89576kB min:5972kB > low:7464kB high:8956kB active_anon:2137168kB inactive_anon:534280kB > active_file:12kB inactive_file:0kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:3288224kB mlocked:0kB dirty:0kB writeback:0kB > mapped:0kB shmem:0kB slab_reclaimable:2204kB slab_unreclaimable:9540kB > kernel_stack:176kB pagetables:5044kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:6 all_unreclaimable? no > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 20957 20957 > Sep 25 11:37:30 ct2 kernel: Node 0 Normal free:38944kB min:38996kB > low:48744kB high:58492kB active_anon:19871956kB inactive_anon:1419396kB > active_file:4kB inactive_file:100kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:21460480kB mlocked:0kB dirty:8kB writeback:0kB > mapped:4kB shmem:0kB slab_reclaimable:19504kB slab_unreclaimable:81520kB > kernel_stack:15768kB pagetables:108956kB unstable:0kB bounce:0kB > writeback_tmp:0kB pages_scanned:156 all_unreclaimable? yes > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 > Sep 25 11:37:30 ct2 kernel: Node 1 Normal free:44996kB min:45108kB > low:56384kB high:67660kB active_anon:22939032kB inactive_anon:1529068kB > active_file:3908kB inactive_file:4280kB unevictable:0kB isolated(anon):0kB > isolated(file):0kB present:24821760kB mlocked:0kB dirty:264kB > writeback:3012kB mapped:4064kB shmem:4kB slab_reclaimable:19944kB > slab_unreclaimable:52480kB kernel_stack:7968kB pagetables:43504kB > unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12544 > all_unreclaimable? yes > Sep 25 11:37:30 ct2 kernel: lowmem_reserve[]: 0 0 0 0 > Sep 25 11:37:30 ct2 kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB > 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB > Sep 25 11:37:30 ct2 kernel: Node 0 DMA32: 638*4kB 601*8kB 469*16kB > 384*32kB 260*64kB 159*128kB 67*256kB 16*512kB 0*1024kB 0*2048kB 0*4096kB = > 89488kB > Sep 25 11:37:30 ct2 kernel: Node 0 Normal: 9695*4kB 0*8kB 0*16kB 0*32kB > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 38780kB > Sep 25 11:37:30 ct2 kernel: Node 1 Normal: 11291*4kB 55*8kB 19*16kB 1*32kB > 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45940kB > Sep 25 11:37:30 ct2 kernel: 10737 total pagecache pages > Sep 25 11:37:30 ct2 kernel: 8679 pages in swap cache > Sep 25 11:37:30 ct2 kernel: Swap cache stats: add 6400017, delete 6391338, > find 89617/105666 > Sep 25 11:37:30 ct2 kernel: Free swap = 0kB > Sep 25 11:37:30 ct2 kernel: Total swap = 24756216kB > Sep 25 11:37:30 ct2 kernel: 12582911 pages RAM > Sep 25 11:37:30 ct2 kernel: 239403 pages reserved > Sep 25 11:37:30 ct2 kernel: 9358 pages shared > Sep 25 11:37:30 ct2 kernel: 12281029 pages non-shared > Sep 25 11:37:30 ct2 kernel: [ pid ] uid tgid total_vm rss cpu > oom_adj oom_score_adj name > Sep 25 11:37:30 ct2 kernel: [ 727] 0 727 2791 0 2 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [ 1590] 0 1590 63856 38 1 > 0 0 rsyslogd > Sep 25 11:37:30 ct2 kernel: [ 1737] 0 1737 20318 24 6 > 0 0 master > Sep 25 11:37:30 ct2 kernel: [ 1744] 89 1744 20381 19 0 > 0 0 qmgr > Sep 25 11:37:30 ct2 kernel: [ 1747] 0 1747 29333 24 0 > 0 0 crond > Sep 25 11:37:30 ct2 kernel: [ 1760] 0 1760 14211 1 0 > 0 0 login > Sep 25 11:37:30 ct2 kernel: [ 1762] 0 1762 1016 1 5 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1764] 0 1764 1016 1 3 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1766] 0 1766 1016 1 0 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1768] 0 1768 1016 1 6 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1770] 0 1770 1016 1 4 > 0 0 mingetty > Sep 25 11:37:30 ct2 kernel: [ 1784] 0 1784 27076 25 5 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [ 4915] 0 4915 2280 1 4 > 0 0 dhclient > Sep 25 11:37:30 ct2 kernel: [ 4958] 0 4958 25089 37 2 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [ 4962] 0 4962 27101 12 3 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [ 5025] 0 5025 16651 0 4 > -17 -1000 sshd > Sep 25 11:37:30 ct2 kernel: [ 5688] 0 5688 6910 10 1 > -17 -1000 auditd > Sep 25 11:37:30 ct2 kernel: [17539] 0 17539 25266 0 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17540] 0 17540 20144 0 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17542] 0 17542 23264 1 5 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17543] 0 17543 39609 1 1 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17932] 0 17932 39609 1 3 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17937] 0 17937 39603 1 2 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [17940] 0 17940 40123 1 0 > 0 0 sfcbd > Sep 25 11:37:30 ct2 kernel: [18632] 0 18632 291742 269 2 > 0 0 dsm_sa_datamgrd > Sep 25 11:37:30 ct2 kernel: [18780] 0 18780 73312 86 4 > 0 0 dsm_sa_eventmgr > Sep 25 11:37:30 ct2 kernel: [18796] 0 18796 171833 21 0 > 0 0 dsm_sa_datamgrd > Sep 25 11:37:30 ct2 kernel: [18842] 0 18842 109528 108 4 > 0 0 dsm_sa_snmpd > Sep 25 11:37:30 ct2 kernel: [19147] 0 19147 33251 1 0 > 0 0 dsm_om_connsvcd > Sep 25 11:37:30 ct2 kernel: [19148] 0 19148 831391 28062 1 > 0 0 dsm_om_connsvcd > Sep 25 11:37:30 ct2 kernel: [38380] 0 38380 25091 4 4 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [38386] 0 38386 27076 4 1 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [28907] 0 28907 2883 0 5 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [28908] 0 28908 2856 0 7 > -17 -1000 udevd > Sep 25 11:37:30 ct2 kernel: [44087] 0 44087 24573 7 0 > 0 0 sshd > Sep 25 11:37:30 ct2 kernel: [44090] 0 44090 27119 8 1 > 0 0 bash > Sep 25 11:37:30 ct2 kernel: [34590] 38 34590 6624 34 2 > 0 0 ntpd > Sep 25 11:37:30 ct2 kernel: [34920] 0 34920 84080 11471 3 > 0 0 ceph-mon > Sep 25 11:37:30 ct2 kernel: [35674] 0 35674 268185 42473 1 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [43044] 0 43044 247471 39531 5 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [ 7125] 0 7125 270942 41559 1 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [20854] 0 20854 251666 40220 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [21799] 0 21799 250641 29693 5 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [22800] 0 22800 275004 33343 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [23509] 0 23509 262113 28210 4 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [24445] 0 24445 259672 33212 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [25146] 0 25146 266437 35462 2 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [25889] 0 25889 257854 34321 0 > 0 0 ceph-osd > Sep 25 11:37:30 ct2 kernel: [27977] 0 27977 166624 482 2 > 0 0 python > Sep 25 11:37:30 ct2 kernel: [28216] 0 28216 1918 12 6 > 0 0 nc > Sep 25 11:37:30 ct2 kernel: [28636] 89 28636 20344 127 4 > 0 0 pickup > > > 2014-09-26 15:48 GMT+04:00 Irek Fasikhov <malmyzh at gmail.com>: > >> OS: CentOS 6.5 >> Kernel: 2.6.32-431.el6.x86_64 >> Ceph --version: ceph version 0.72.2 >> (a913ded2ff138aefb8cb84d347d72164099cfd60) >> >> >> >> 2014-09-26 15:44 GMT+04:00 Irek Fasikhov <malmyzh at gmail.com>: >> >>> Hi, All. >>> >>> I see a memory leak when importing raw devi?e. >>> >>> Export Scheme: >>> [rbd at rbdbackup ~]$ rbd --no-progress -n client.rbdbackup -k >>> /etc/ceph/big.keyring -c /etc/ceph/big.conf export rbdtest/vm-111-disk-1 - >>> | nc 10.43.255.252 12345 >>> >>> [root at ct2 ~]# nc -l 12345 | rbd import --no-progress --image-format 2 - >>> rbd/vm-111-disk-1 >>> >>> This is the same problem with ssh >>> >>> Memory usage, see the screenshots: >>> >>> https://drive.google.com/folderview?id=0BxoNLVWxzOJWSHlTSEZvM3lkQXM&usp=sharing >>> >>> -- >>> ? ?????????, ??????? ???? ??????????? >>> ???.: +79229045757 >>> >> >> >> >> -- >> ? ?????????, ??????? ???? ??????????? >> ???.: +79229045757 >> > > > > -- > ? ?????????, ??????? ???? ??????????? > ???.: +79229045757 > -- ? ?????????, ??????? ???? ??????????? ???.: +79229045757 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140926/5201e5b8/attachment.htm>
 |