I found some memory leak. could you please try https://github.com/ceph/ceph/pull/22240 On Fri, May 25, 2018 at 1:49 PM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote: > Here the result: > > > root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net flush journal > { > "message": "", > "return_code": 0 > } > root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net config set mds_cache_size 10000 > { > "success": "mds_cache_size = '10000' (not observed, change may require restart) " > } > > wait ... > > > root@ceph4-2:~# ceph tell mds.ceph4-2.odiso.net heap stats > 2018-05-25 07:44:02.185911 7f4cad7fa700 0 client.50748489 ms_handle_reset on 10.5.0.88:6804/994206868 > 2018-05-25 07:44:02.196160 7f4cae7fc700 0 client.50792764 ms_handle_reset on 10.5.0.88:6804/994206868 > mds.ceph4-2.odiso.net tcmalloc heap stats:------------------------------------------------ > MALLOC: 13175782328 (12565.4 MiB) Bytes in use by application > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > MALLOC: + 1774628488 ( 1692.4 MiB) Bytes in central cache freelist > MALLOC: + 34274608 ( 32.7 MiB) Bytes in transfer cache freelist > MALLOC: + 57260176 ( 54.6 MiB) Bytes in thread cache freelists > MALLOC: + 120582336 ( 115.0 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 15162527936 (14460.1 MiB) Actual memory used (physical + swap) > MALLOC: + 4974067712 ( 4743.6 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 20136595648 (19203.8 MiB) Virtual address space used > MALLOC: > MALLOC: 1852388 Spans in use > MALLOC: 18 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). > Bytes released to the OS take up virtual address space but no physical memory. > > > root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net config set mds_cache_size 0 > { > "success": "mds_cache_size = '0' (not observed, change may require restart) " > } > > ----- Mail original ----- > De: "Zheng Yan" <ukernel@xxxxxxxxx> > À: "aderumier" <aderumier@xxxxxxxxx> > Envoyé: Vendredi 25 Mai 2018 05:56:31 > Objet: Re: ceph mds memory usage 20GB : is it normal ? > > On Thu, May 24, 2018 at 11:34 PM, Alexandre DERUMIER > <aderumier@xxxxxxxxx> wrote: >>>>Still don't find any clue. Does the cephfs have idle period. If it >>>>has, could you decrease mds's cache size and check what happens. For >>>>example, run following commands during the old period. >> >>>>ceph daemon mds.xx flush journal >>>>ceph daemon mds.xx config set mds_cache_size 10000; >>>>"wait a minute" >>>>ceph tell mds.xx heap stats >>>>ceph daemon mds.xx config set mds_cache_size 0 >> >> ok thanks. I'll try this night. >> >> I have already mds_cache_memory_limit = 5368709120, >> >> does it need to remove it first before setting mds_cache_size 10000 ? > > no >> >> >> >> >> ----- Mail original ----- >> De: "Zheng Yan" <ukernel@xxxxxxxxx> >> À: "aderumier" <aderumier@xxxxxxxxx> >> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >> Envoyé: Jeudi 24 Mai 2018 16:27:21 >> Objet: Re: ceph mds memory usage 20GB : is it normal ? >> >> On Thu, May 24, 2018 at 7:22 PM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote: >>> Thanks! >>> >>> >>> here the profile.pdf >>> >>> 10-15min profiling, I can't do it longer because my clients where lagging. >>> >>> but I think it should be enough to observe the rss memory increase. >>> >>> >> >> Still don't find any clue. Does the cephfs have idle period. If it >> has, could you decrease mds's cache size and check what happens. For >> example, run following commands during the old period. >> >> ceph daemon mds.xx flush journal >> ceph daemon mds.xx config set mds_cache_size 10000; >> "wait a minute" >> ceph tell mds.xx heap stats >> ceph daemon mds.xx config set mds_cache_size 0 >> >> >>> >>> >>> ----- Mail original ----- >>> De: "Zheng Yan" <ukernel@xxxxxxxxx> >>> À: "aderumier" <aderumier@xxxxxxxxx> >>> Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>> Envoyé: Jeudi 24 Mai 2018 11:34:20 >>> Objet: Re: ceph mds memory usage 20GB : is it normal ? >>> >>> On Tue, May 22, 2018 at 3:11 PM, Alexandre DERUMIER <aderumier@xxxxxxxxx> wrote: >>>> Hi,some new stats, mds memory is not 16G, >>>> >>>> I have almost same number of items and bytes in cache vs some weeks ago when mds was using 8G. (ceph 12.2.5) >>>> >>>> >>>> root@ceph4-2:~# while sleep 1; do ceph daemon mds.ceph4-2.odiso.net perf dump | jq '.mds_mem.rss'; ceph daemon mds.ceph4-2.odiso.net dump_mempools | jq -c '.mds_co'; done >>>> 16905052 >>>> {"items":43350988,"bytes":5257428143} >>>> 16905052 >>>> {"items":43428329,"bytes":5283850173} >>>> 16905052 >>>> {"items":43209167,"bytes":5208578149} >>>> 16905052 >>>> {"items":43177631,"bytes":5198833577} >>>> 16905052 >>>> {"items":43312734,"bytes":5252649462} >>>> 16905052 >>>> {"items":43355753,"bytes":5277197972} >>>> 16905052 >>>> {"items":43700693,"bytes":5303376141} >>>> 16905052 >>>> {"items":43115809,"bytes":5156628138} >>>> ^C >>>> >>>> >>>> >>>> >>>> root@ceph4-2:~# ceph status >>>> cluster: >>>> id: e22b8e83-3036-4fe5-8fd5-5ce9d539beca >>>> health: HEALTH_OK >>>> >>>> services: >>>> mon: 3 daemons, quorum ceph4-1,ceph4-2,ceph4-3 >>>> mgr: ceph4-1.odiso.net(active), standbys: ceph4-2.odiso.net, ceph4-3.odiso.net >>>> mds: cephfs4-1/1/1 up {0=ceph4-2.odiso.net=up:active}, 2 up:standby >>>> osd: 18 osds: 18 up, 18 in >>>> rgw: 3 daemons active >>>> >>>> data: >>>> pools: 11 pools, 1992 pgs >>>> objects: 75677k objects, 6045 GB >>>> usage: 20579 GB used, 6246 GB / 26825 GB avail >>>> pgs: 1992 active+clean >>>> >>>> io: >>>> client: 14441 kB/s rd, 2550 kB/s wr, 371 op/s rd, 95 op/s wr >>>> >>>> >>>> root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net cache status >>>> { >>>> "pool": { >>>> "items": 44523608, >>>> "bytes": 5326049009 >>>> } >>>> } >>>> >>>> >>>> root@ceph4-2:~# ceph daemon mds.ceph4-2.odiso.net perf dump >>>> { >>>> "AsyncMessenger::Worker-0": { >>>> "msgr_recv_messages": 798876013, >>>> "msgr_send_messages": 825999506, >>>> "msgr_recv_bytes": 7003223097381, >>>> "msgr_send_bytes": 691501283744, >>>> "msgr_created_connections": 148, >>>> "msgr_active_connections": 146, >>>> "msgr_running_total_time": 39914.832387470, >>>> "msgr_running_send_time": 13744.704199430, >>>> "msgr_running_recv_time": 32342.160588451, >>>> "msgr_running_fast_dispatch_time": 5996.336446782 >>>> }, >>>> "AsyncMessenger::Worker-1": { >>>> "msgr_recv_messages": 429668771, >>>> "msgr_send_messages": 414760220, >>>> "msgr_recv_bytes": 5003149410825, >>>> "msgr_send_bytes": 396281427789, >>>> "msgr_created_connections": 132, >>>> "msgr_active_connections": 132, >>>> "msgr_running_total_time": 23644.410515392, >>>> "msgr_running_send_time": 7669.068710688, >>>> "msgr_running_recv_time": 19751.610043696, >>>> "msgr_running_fast_dispatch_time": 4331.023453385 >>>> }, >>>> "AsyncMessenger::Worker-2": { >>>> "msgr_recv_messages": 1312910919, >>>> "msgr_send_messages": 1260040403, >>>> "msgr_recv_bytes": 5330386980976, >>>> "msgr_send_bytes": 3341965016878, >>>> "msgr_created_connections": 143, >>>> "msgr_active_connections": 138, >>>> "msgr_running_total_time": 61696.635450100, >>>> "msgr_running_send_time": 23491.027014598, >>>> "msgr_running_recv_time": 53858.409319734, >>>> "msgr_running_fast_dispatch_time": 4312.451966809 >>>> }, >>>> "finisher-PurgeQueue": { >>>> "queue_len": 0, >>>> "complete_latency": { >>>> "avgcount": 1889416, >>>> "sum": 29224.227703697, >>>> "avgtime": 0.015467333 >>>> } >>>> }, >>>> "mds": { >>>> "request": 1822420924, >>>> "reply": 1822420886, >>>> "reply_latency": { >>>> "avgcount": 1822420886, >>>> "sum": 5258467.616943274, >>>> "avgtime": 0.002885429 >>>> }, >>>> "forward": 0, >>>> "dir_fetch": 116035485, >>>> "dir_commit": 1865012, >>>> "dir_split": 17, >>>> "dir_merge": 24, >>>> "inode_max": 2147483647, >>>> "inodes": 1600438, >>>> "inodes_top": 210492, >>>> "inodes_bottom": 100560, >>>> "inodes_pin_tail": 1289386, >>>> "inodes_pinned": 1299735, >>>> "inodes_expired": 22223476046, >>>> "inodes_with_caps": 1299137, >>>> "caps": 2211546, >>>> "subtrees": 2, >>>> "traverse": 1953482456, >>>> "traverse_hit": 1127647211, >>>> "traverse_forward": 0, >>>> "traverse_discover": 0, >>>> "traverse_dir_fetch": 105833969, >>>> "traverse_remote_ino": 31686, >>>> "traverse_lock": 4344, >>>> "load_cent": 182244014474, >>>> "q": 104, >>>> "exported": 0, >>>> "exported_inodes": 0, >>>> "imported": 0, >>>> "imported_inodes": 0 >>>> }, >>>> "mds_cache": { >>>> "num_strays": 14980, >>>> "num_strays_delayed": 7, >>>> "num_strays_enqueuing": 0, >>>> "strays_created": 1672815, >>>> "strays_enqueued": 1659514, >>>> "strays_reintegrated": 666, >>>> "strays_migrated": 0, >>>> "num_recovering_processing": 0, >>>> "num_recovering_enqueued": 0, >>>> "num_recovering_prioritized": 0, >>>> "recovery_started": 2, >>>> "recovery_completed": 2, >>>> "ireq_enqueue_scrub": 0, >>>> "ireq_exportdir": 0, >>>> "ireq_flush": 0, >>>> "ireq_fragmentdir": 41, >>>> "ireq_fragstats": 0, >>>> "ireq_inodestats": 0 >>>> }, >>>> "mds_log": { >>>> "evadd": 357717092, >>>> "evex": 357717106, >>>> "evtrm": 357716741, >>>> "ev": 105198, >>>> "evexg": 0, >>>> "evexd": 365, >>>> "segadd": 437124, >>>> "segex": 437124, >>>> "segtrm": 437123, >>>> "seg": 130, >>>> "segexg": 0, >>>> "segexd": 1, >>>> "expos": 6916004026339, >>>> "wrpos": 6916179441942, >>>> "rdpos": 6319502327537, >>>> "jlat": { >>>> "avgcount": 59071693, >>>> "sum": 120823.311894779, >>>> "avgtime": 0.002045367 >>>> }, >>>> "replayed": 104847 >>>> }, >>>> "mds_mem": { >>>> "ino": 1599422, >>>> "ino+": 22152405695, >>>> "ino-": 22150806273, >>>> "dir": 256943, >>>> "dir+": 18460298, >>>> "dir-": 18203355, >>>> "dn": 1600689, >>>> "dn+": 22227888283, >>>> "dn-": 22226287594, >>>> "cap": 2211546, >>>> "cap+": 1674287476, >>>> "cap-": 1672075930, >>>> "rss": 16905052, >>>> "heap": 313916, >>>> "buf": 0 >>>> }, >>>> "mds_server": { >>>> "dispatch_client_request": 1964131912, >>>> "dispatch_server_request": 0, >>>> "handle_client_request": 1822420924, >>>> "handle_client_session": 15557609, >>>> "handle_slave_request": 0, >>>> "req_create": 4116952, >>>> "req_getattr": 18696543, >>>> "req_getfilelock": 0, >>>> "req_link": 6625, >>>> "req_lookup": 1425824734, >>>> "req_lookuphash": 0, >>>> "req_lookupino": 0, >>>> "req_lookupname": 8703, >>>> "req_lookupparent": 0, >>>> "req_lookupsnap": 0, >>>> "req_lssnap": 0, >>>> "req_mkdir": 371878, >>>> "req_mknod": 0, >>>> "req_mksnap": 0, >>>> "req_open": 351119806, >>>> "req_readdir": 17103599, >>>> "req_rename": 2437529, >>>> "req_renamesnap": 0, >>>> "req_rmdir": 78789, >>>> "req_rmsnap": 0, >>>> "req_rmxattr": 0, >>>> "req_setattr": 4547650, >>>> "req_setdirlayout": 0, >>>> "req_setfilelock": 633219, >>>> "req_setlayout": 0, >>>> "req_setxattr": 2, >>>> "req_symlink": 2520, >>>> "req_unlink": 1589288 >>>> }, >>>> "mds_sessions": { >>>> "session_count": 321, >>>> "session_add": 383, >>>> "session_remove": 62 >>>> }, >>>> "objecter": { >>>> "op_active": 0, >>>> "op_laggy": 0, >>>> "op_send": 197932443, >>>> "op_send_bytes": 605992324653, >>>> "op_resend": 22, >>>> "op_reply": 197932421, >>>> "op": 197932421, >>>> "op_r": 116256030, >>>> "op_w": 81676391, >>>> "op_rmw": 0, >>>> "op_pg": 0, >>>> "osdop_stat": 1518341, >>>> "osdop_create": 4314348, >>>> "osdop_read": 79810, >>>> "osdop_write": 59151421, >>>> "osdop_writefull": 237358, >>>> "osdop_writesame": 0, >>>> "osdop_append": 0, >>>> "osdop_zero": 2, >>>> "osdop_truncate": 9, >>>> "osdop_delete": 2320714, >>>> "osdop_mapext": 0, >>>> "osdop_sparse_read": 0, >>>> "osdop_clonerange": 0, >>>> "osdop_getxattr": 29426577, >>>> "osdop_setxattr": 8628696, >>>> "osdop_cmpxattr": 0, >>>> "osdop_rmxattr": 0, >>>> "osdop_resetxattrs": 0, >>>> "osdop_tmap_up": 0, >>>> "osdop_tmap_put": 0, >>>> "osdop_tmap_get": 0, >>>> "osdop_call": 0, >>>> "osdop_watch": 0, >>>> "osdop_notify": 0, >>>> "osdop_src_cmpxattr": 0, >>>> "osdop_pgls": 0, >>>> "osdop_pgls_filter": 0, >>>> "osdop_other": 13551599, >>>> "linger_active": 0, >>>> "linger_send": 0, >>>> "linger_resend": 0, >>>> "linger_ping": 0, >>>> "poolop_active": 0, >>>> "poolop_send": 0, >>>> "poolop_resend": 0, >>>> "poolstat_active": 0, >>>> "poolstat_send": 0, >>>> "poolstat_resend": 0, >>>> "statfs_active": 0, >>>> "statfs_send": 0, >>>> "statfs_resend": 0, >>>> "command_active": 0, >>>> "command_send": 0, >>>> "command_resend": 0, >>>> "map_epoch": 3907, >>>> "map_full": 0, >>>> "map_inc": 601, >>>> "osd_sessions": 18, >>>> "osd_session_open": 20, >>>> "osd_session_close": 2, >>>> "osd_laggy": 0, >>>> "omap_wr": 3595801, >>>> "omap_rd": 232070972, >>>> "omap_del": 272598 >>>> }, >>>> "purge_queue": { >>>> "pq_executing_ops": 0, >>>> "pq_executing": 0, >>>> "pq_executed": 1659514 >>>> }, >>>> "throttle-msgr_dispatch_throttler-mds": { >>>> "val": 0, >>>> "max": 104857600, >>>> "get_started": 0, >>>> "get": 2541455703, >>>> "get_sum": 17148691767160, >>>> "get_or_fail_fail": 0, >>>> "get_or_fail_success": 2541455703, >>>> "take": 0, >>>> "take_sum": 0, >>>> "put": 2541455703, >>>> "put_sum": 17148691767160, >>>> "wait": { >>>> "avgcount": 0, >>>> "sum": 0.000000000, >>>> "avgtime": 0.000000000 >>>> } >>>> }, >>>> "throttle-objecter_bytes": { >>>> "val": 0, >>>> "max": 104857600, >>>> "get_started": 0, >>>> "get": 0, >>>> "get_sum": 0, >>>> "get_or_fail_fail": 0, >>>> "get_or_fail_success": 0, >>>> "take": 197932421, >>>> "take_sum": 606323353310, >>>> "put": 182060027, >>>> "put_sum": 606323353310, >>>> "wait": { >>>> "avgcount": 0, >>>> "sum": 0.000000000, >>>> "avgtime": 0.000000000 >>>> } >>>> }, >>>> "throttle-objecter_ops": { >>>> "val": 0, >>>> "max": 1024, >>>> "get_started": 0, >>>> "get": 0, >>>> "get_sum": 0, >>>> "get_or_fail_fail": 0, >>>> "get_or_fail_success": 0, >>>> "take": 197932421, >>>> "take_sum": 197932421, >>>> "put": 197932421, >>>> "put_sum": 197932421, >>>> "wait": { >>>> "avgcount": 0, >>>> "sum": 0.000000000, >>>> "avgtime": 0.000000000 >>>> } >>>> }, >>>> "throttle-write_buf_throttle": { >>>> "val": 0, >>>> "max": 3758096384, >>>> "get_started": 0, >>>> "get": 1659514, >>>> "get_sum": 154334946, >>>> "get_or_fail_fail": 0, >>>> "get_or_fail_success": 1659514, >>>> "take": 0, >>>> "take_sum": 0, >>>> "put": 79728, >>>> "put_sum": 154334946, >>>> "wait": { >>>> "avgcount": 0, >>>> "sum": 0.000000000, >>>> "avgtime": 0.000000000 >>>> } >>>> }, >>>> "throttle-write_buf_throttle-0x55decea8e140": { >>>> "val": 255839, >>>> "max": 3758096384, >>>> "get_started": 0, >>>> "get": 357717092, >>>> "get_sum": 596677113363, >>>> "get_or_fail_fail": 0, >>>> "get_or_fail_success": 357717092, >>>> "take": 0, >>>> "take_sum": 0, >>>> "put": 59071693, >>>> "put_sum": 596676857524, >>>> "wait": { >>>> "avgcount": 0, >>>> "sum": 0.000000000, >>>> "avgtime": 0.000000000 >>>> } >>>> } >>>> } >>>> >>>> >>> >>> Maybe there is memory leak. What is output of 'ceph tell mds.xx heap >>> stats'. If the RSS size keeps increasing, please run profile heap for >>> a period of time. >>> >>> >>> ceph tell mds.xx heap start_profiler >>> "wait some time" >>> ceph tell mds.xx heap dump >>> ceph tell mds.xx heap stop_profiler >>> pprof --pdf <location pf ceph-mds binary> >>> /var/log/ceph/mds.xxx.profile.* > profile.pdf >>> >>> send profile.pdf to us >>> >>> Regards >>> Yan, Zheng >>> >>>> >>>> ----- Mail original ----- >>>> De: "Webert de Souza Lima" <webert.boss@xxxxxxxxx> >>>> À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> >>>> Envoyé: Lundi 14 Mai 2018 15:14:35 >>>> Objet: Re: ceph mds memory usage 20GB : is it normal ? >>>> >>>> On Sat, May 12, 2018 at 3:11 AM Alexandre DERUMIER < [ mailto:aderumier@xxxxxxxxx | aderumier@xxxxxxxxx ] > wrote: >>>> >>>> >>>> The documentation (luminous) say: >>>> >>>> >>>> >>>> >>>> >>>>>mds cache size >>>>> >>>>>Description: The number of inodes to cache. A value of 0 indicates an unlimited number. It is recommended to use mds_cache_memory_limit to limit the amount of memory the MDS cache uses. >>>>>Type: 32-bit Integer >>>>>Default: 0 >>>>> >>>> >>>> >>>> >>>> and, my mds_cache_memory_limit is currently at 5GB. >>>> >>>> >>>> yeah I have only suggested that because the high memory usage seemed to trouble you and it might be a bug, so it's more of a workaround. >>>> >>>> Regards, >>>> Webert Lima >>>> DevOps Engineer at MAV Tecnologia >>>> Belo Horizonte - Brasil >>>> IRC NICK - WebertRLZ >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |