Hi Matt, Thanks you reply! These days I add memory profiling code to RGW. Because valgrind will slow down radosgw speed, resulting in no recurrence The following is the result after the test. ceph daemon /var/run/ceph/client.radosgw.An1H3c.4010609.140718309098104.asok heap dump client.radosgw.An1H3c dumping heap profile now. ------------------------------------------------ MALLOC: 76501704 ( 73.0 MiB) Bytes in use by application MALLOC: + 3817717760 ( 3640.9 MiB) Bytes in page heap freelist MALLOC: + 21446160 ( 20.5 MiB) Bytes in central cache freelist MALLOC: + 1908224 ( 1.8 MiB) Bytes in transfer cache freelist MALLOC: + 47477032 ( 45.3 MiB) Bytes in thread cache freelists MALLOC: + 9437184 ( 9.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3974488064 ( 3790.4 MiB) Actual memory used (physical + swap) MALLOC: + 1202749440 ( 1147.0 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 5177237504 ( 4937.4 MiB) Virtual address space used MALLOC: MALLOC: 10040 Spans in use MALLOC: 587 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. I could use command 'heap release' to set page heap freelist is 0 MiB. So the RGW memory is down from 'top -p ' command.It look like tcmalloc problem. What do you think of it？ WeiGuo Matt Benjamin <mbenjami@xxxxxxxxxx> 于2020年12月10日周四 下午9:41写道： > > Hi WeiGuo, > > I'm not aware of a memory leak in RGW, but in the past when we've > suspected one, we used valgrind massif against radosgw for a > representative time sample to get a picture of memory behavior in the > program. That might be a way to start. > > Matt > > On Thu, Dec 10, 2020 at 8:14 AM WeiGuo Ren <rwg1335252904@xxxxxxxxx> wrote: > > > > In my ceph version 14.2.5.ceph radosgw mem is high (4.6g use top > > command).when I write 20m. > > I noticed dump_mempools's buffer_anon is very high (3-4g)in > > testing,after test,the buffer_anon has been release(about10m). but rgw > > mem is still high(about 4.6g).So i think maybe some memleak occurred > > in rgw. > > Could someone can help me? or Could someone tell me how to tune rgw-memory? > > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 >
