On Wed, Mar 21, 2018 at 2:26 PM, Kjetil Joergensen <kjetil@xxxxxxxxxxxx> wrote: > I retract my previous statement(s). > > My current suspicion is that this isn't a leak as much as it being > load-driven, after enough waiting - it generally seems to settle around some > equilibrium. We do seem to sit on the mempools x 2.4 ~ ceph-osd RSS, which > is on the higher side (I see documentation alluding to expecting ~1.5x). > > -KJ > > On Mon, Mar 19, 2018 at 3:05 AM, Konstantin Shalygin <k0ste@xxxxxxxx> wrote: >> >> >>> We don't run compression as far as I know, so that wouldn't be it. We do >>> actually run a mix of bluestore & filestore - due to the rest of the >>> cluster predating a stable bluestore by some amount. >> >> >> >> 12.2.2 -> 12.2.4 at 2018/03/10: I don't see increase of memory usage. No >> any compressions of course. >> >> >> >> http://storage6.static.itmages.com/i/18/0319/h_1521453809_9131482_859b1fb0a5.png >> I am seeing these entries under load - should be plenty of RAM on a node with 128GB RAM and 18 OSDs Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193331] winbindd cpuset=/ mems_allowed=0-1 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193337] CPU: 3 PID: 3406 Comm: winbindd Not tainted 4.14.14-041414-generic #201801201219 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193338] Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193339] Call Trace: Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193347] dump_stack+0x5c/0x85 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193351] dump_header+0x94/0x229 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193355] ? do_try_to_free_pages+0x2a1/0x330 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193357] ? get_page_from_freelist+0xa3/0xb20 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193359] oom_kill_process+0x213/0x410 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193361] out_of_memory+0x2af/0x4d0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193363] __alloc_pages_slowpath+0xab2/0xe40 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193366] __alloc_pages_nodemask+0x261/0x280 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193370] filemap_fault+0x33f/0x6b0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193373] ? filemap_map_pages+0x18a/0x3a0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193376] ext4_filemap_fault+0x2c/0x40 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193379] __do_fault+0x19/0xe0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193381] __handle_mm_fault+0xcd6/0x1180 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193383] handle_mm_fault+0xaa/0x1f0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193387] __do_page_fault+0x25d/0x4e0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193391] ? page_fault+0x36/0x60 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193393] page_fault+0x4c/0x60 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193396] RIP: 0033:0x56443d3d1239 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193397] RSP: 002b:00007ffe6e44b3a0 EFLAGS: 00010246 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193399] Mem-Info: Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] active_anon:30843938 inactive_anon:1403277 isolated_anon:0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] active_file:121 inactive_file:977 isolated_file:18 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] unevictable:3203 dirty:2 writeback:0 unstable:0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] slab_reclaimable:51522 slab_unreclaimable:95924 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] mapped:2926 shmem:5220 pagetables:77204 bounce:0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193407] free:328371 free_pcp:0 free_cma:0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193411] Node 0 active_anon:61155956kB inactive_anon:3014752kB active_file:864kB inactive_file:1432kB unevictable:10440kB isolated(anon):0kB isolated(file):80kB mapped:7648kB dirty:0kB writeback:0kB shmem:14460kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193414] Node 1 active_anon:62219796kB inactive_anon:2598356kB active_file:0kB inactive_file:2476kB unevictable:2372kB isolated(anon):0kB isolated(file):0kB mapped:4056kB dirty:8kB writeback:0kB shmem:6420kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193416] Node 0 DMA free:15896kB min:124kB low:152kB high:180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193420] lowmem_reserve[]: 0 1889 64319 64319 64319 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193424] Node 0 DMA32 free:265308kB min:15732kB low:19664kB high:23596kB active_anon:1642352kB inactive_anon:63060kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2045868kB managed:1980300kB mlocked:0kB kernel_stack:48kB pagetables:832kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193428] lowmem_reserve[]: 0 0 62430 62430 62430 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193432] Node 0 Normal free:507908kB min:507928kB low:634908kB high:761888kB active_anon:59513604kB inactive_anon:2951692kB active_file:732kB inactive_file:1720kB unevictable:10440kB writepending:0kB present:65011712kB managed:63934936kB mlocked:10440kB kernel_stack:16392kB pagetables:164944kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193436] lowmem_reserve[]: 0 0 0 0 0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193440] Node 1 Normal free:524372kB min:524784kB low:655980kB high:787176kB active_anon:62219796kB inactive_anon:2598356kB active_file:504kB inactive_file:1392kB unevictable:2372kB writepending:8kB present:67108864kB managed:66056740kB mlocked:2372kB kernel_stack:17912kB pagetables:143040kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193444] lowmem_reserve[]: 0 0 0 0 0 Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193447] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15896kB Mar 26 07:55:32 roc04r-sc3a085 kernel: [733474.193459] Node 0 DMA32: 403*4kB (UME) 238*8kB (UME) 196*16kB (UME) 102*32kB (UME) 56*64kB (UME) 24*128kB (UE) 25*256kB (UM) 11*512kB (UME) 4*1024kB (UE) 6*2048kB (UM) 54*4096kB (UM) = 266172kB >> >> >> >> k > > > > > -- > Kjetil Joergensen <kjetil@xxxxxxxxxxxx> > SRE, Medallia Inc > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |