Thanks for the advice, we are using Debian 9 (stretch), with a custom Linux kernel 4.14. But "heap release" didn't help. Le lundi 08 avril 2019 à 12:18 +0200, Dan van der Ster a écrit : > Which OS are you using? > With CentOS we find that the heap is not always automatically > released. (You can check the heap freelist with `ceph tell osd.0 heap > stats`). > As a workaround we run this hourly: > > ceph tell mon.* heap release > ceph tell osd.* heap release > ceph tell mds.* heap release > > -- Dan > > On Sat, Apr 6, 2019 at 1:30 PM Olivier Bonvalet <ceph.list@xxxxxxxxx> > wrote: > > Hi, > > > > on a Luminous 12.2.11 deploiement, my bluestore OSD exceed the > > osd_memory_target : > > > > daevel-ob@ssdr712h:~$ ps auxw | grep ceph-osd > > ceph 3646 17.1 12.0 6828916 5893136 ? Ssl mars29 > > 1903:42 /usr/bin/ceph-osd -f --cluster ceph --id 143 --setuser ceph > > --setgroup ceph > > ceph 3991 12.9 11.2 6342812 5485356 ? Ssl mars29 > > 1443:41 /usr/bin/ceph-osd -f --cluster ceph --id 144 --setuser ceph > > --setgroup ceph > > ceph 4361 16.9 11.8 6718432 5783584 ? Ssl mars29 > > 1889:41 /usr/bin/ceph-osd -f --cluster ceph --id 145 --setuser ceph > > --setgroup ceph > > ceph 4731 19.7 12.2 6949584 5982040 ? Ssl mars29 > > 2198:47 /usr/bin/ceph-osd -f --cluster ceph --id 146 --setuser ceph > > --setgroup ceph > > ceph 5073 16.7 11.6 6639568 5701368 ? Ssl mars29 > > 1866:05 /usr/bin/ceph-osd -f --cluster ceph --id 147 --setuser ceph > > --setgroup ceph > > ceph 5417 14.6 11.2 6386764 5519944 ? Ssl mars29 > > 1634:30 /usr/bin/ceph-osd -f --cluster ceph --id 148 --setuser ceph > > --setgroup ceph > > ceph 5760 16.9 12.0 6806448 5879624 ? Ssl mars29 > > 1882:42 /usr/bin/ceph-osd -f --cluster ceph --id 149 --setuser ceph > > --setgroup ceph > > ceph 6105 16.0 11.6 6576336 5694556 ? Ssl mars29 > > 1782:52 /usr/bin/ceph-osd -f --cluster ceph --id 150 --setuser ceph > > --setgroup ceph > > > > daevel-ob@ssdr712h:~$ free -m > > total used free shared buff/cache > > available > > Mem: 47771 45210 1643 17 917 > > 43556 > > Swap: 0 0 0 > > > > # ceph daemon osd.147 config show | grep memory_target > > "osd_memory_target": "4294967296", > > > > > > And there is no recovery / backfilling, the cluster is fine : > > > > $ ceph status > > cluster: > > id: de035250-323d-4cf6-8c4b-cf0faf6296b1 > > health: HEALTH_OK > > > > services: > > mon: 5 daemons, quorum tolriq,tsyne,olkas,lorunde,amphel > > mgr: tsyne(active), standbys: olkas, tolriq, lorunde, amphel > > osd: 120 osds: 116 up, 116 in > > > > data: > > pools: 20 pools, 12736 pgs > > objects: 15.29M objects, 31.1TiB > > usage: 101TiB used, 75.3TiB / 177TiB avail > > pgs: 12732 active+clean > > 4 active+clean+scrubbing+deep > > > > io: > > client: 72.3MiB/s rd, 26.8MiB/s wr, 2.30kop/s rd, > > 1.29kop/s wr > > > > > > On an other host, in the same pool, I see also high memory usage > > : > > > > daevel-ob@ssdr712g:~$ ps auxw | grep ceph-osd > > ceph 6287 6.6 10.6 6027388 5190032 ? Ssl mars21 > > 1511:07 /usr/bin/ceph-osd -f --cluster ceph --id 131 --setuser ceph > > --setgroup ceph > > ceph 6759 7.3 11.2 6299140 5484412 ? Ssl mars21 > > 1665:22 /usr/bin/ceph-osd -f --cluster ceph --id 132 --setuser ceph > > --setgroup ceph > > ceph 7114 7.0 11.7 6576168 5756236 ? Ssl mars21 > > 1612:09 /usr/bin/ceph-osd -f --cluster ceph --id 133 --setuser ceph > > --setgroup ceph > > ceph 7467 7.4 11.1 6244668 5430512 ? Ssl mars21 > > 1704:06 /usr/bin/ceph-osd -f --cluster ceph --id 134 --setuser ceph > > --setgroup ceph > > ceph 7821 7.7 11.1 6309456 5469376 ? Ssl mars21 > > 1754:35 /usr/bin/ceph-osd -f --cluster ceph --id 135 --setuser ceph > > --setgroup ceph > > ceph 8174 6.9 11.6 6545224 5705412 ? Ssl mars21 > > 1590:31 /usr/bin/ceph-osd -f --cluster ceph --id 136 --setuser ceph > > --setgroup ceph > > ceph 8746 6.6 11.1 6290004 5477204 ? Ssl mars21 > > 1511:11 /usr/bin/ceph-osd -f --cluster ceph --id 137 --setuser ceph > > --setgroup ceph > > ceph 9100 7.7 11.6 6552080 5713560 ? Ssl mars21 > > 1757:22 /usr/bin/ceph-osd -f --cluster ceph --id 138 --setuser ceph > > --setgroup ceph > > > > But ! On a similar host, in a different pool, the problem is > > less visible : > > > > daevel-ob@ssdr712i:~$ ps auxw | grep ceph-osd > > ceph 3617 2.8 9.9 5660308 4847444 ? Ssl mars29 > > 313:05 /usr/bin/ceph-osd -f --cluster ceph --id 151 --setuser ceph > > --setgroup ceph > > ceph 3958 2.3 9.8 5661936 4834320 ? Ssl mars29 > > 256:55 /usr/bin/ceph-osd -f --cluster ceph --id 152 --setuser ceph > > --setgroup ceph > > ceph 4299 2.3 9.8 5620616 4807248 ? Ssl mars29 > > 266:26 /usr/bin/ceph-osd -f --cluster ceph --id 153 --setuser ceph > > --setgroup ceph > > ceph 4643 2.3 9.6 5527724 4713572 ? Ssl mars29 > > 262:50 /usr/bin/ceph-osd -f --cluster ceph --id 154 --setuser ceph > > --setgroup ceph > > ceph 5016 2.2 9.7 5597504 4783412 ? Ssl mars29 > > 248:37 /usr/bin/ceph-osd -f --cluster ceph --id 155 --setuser ceph > > --setgroup ceph > > ceph 5380 2.8 9.9 5700204 4886432 ? Ssl mars29 > > 321:05 /usr/bin/ceph-osd -f --cluster ceph --id 156 --setuser ceph > > --setgroup ceph > > ceph 5724 3.1 10.1 5767456 4953484 ? Ssl mars29 > > 352:55 /usr/bin/ceph-osd -f --cluster ceph --id 157 --setuser ceph > > --setgroup ceph > > ceph 6070 2.7 9.9 5683092 4868632 ? Ssl mars29 > > 309:10 /usr/bin/ceph-osd -f --cluster ceph --id 158 --setuser ceph > > --setgroup ceph > > > > > > Is there some memory leak ? Or should I expect that > > osd_memory_target > > (the default 4GB here) is not really followed, and so reduce it > > ? > > > > Thanks, > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |