Hi,
on a Luminous 12.2.11 deploiement, my bluestore OSD exceed the
osd_memory_target :
daevel-ob@ssdr712h:~$ ps auxw | grep ceph-osd
ceph 3646 17.1 12.0 6828916 5893136 ? Ssl mars29 1903:42 /usr/bin/ceph-osd -f --cluster ceph --id 143 --setuser ceph --setgroup ceph
ceph 3991 12.9 11.2 6342812 5485356 ? Ssl mars29 1443:41 /usr/bin/ceph-osd -f --cluster ceph --id 144 --setuser ceph --setgroup ceph
ceph 4361 16.9 11.8 6718432 5783584 ? Ssl mars29 1889:41 /usr/bin/ceph-osd -f --cluster ceph --id 145 --setuser ceph --setgroup ceph
ceph 4731 19.7 12.2 6949584 5982040 ? Ssl mars29 2198:47 /usr/bin/ceph-osd -f --cluster ceph --id 146 --setuser ceph --setgroup ceph
ceph 5073 16.7 11.6 6639568 5701368 ? Ssl mars29 1866:05 /usr/bin/ceph-osd -f --cluster ceph --id 147 --setuser ceph --setgroup ceph
ceph 5417 14.6 11.2 6386764 5519944 ? Ssl mars29 1634:30 /usr/bin/ceph-osd -f --cluster ceph --id 148 --setuser ceph --setgroup ceph
ceph 5760 16.9 12.0 6806448 5879624 ? Ssl mars29 1882:42 /usr/bin/ceph-osd -f --cluster ceph --id 149 --setuser ceph --setgroup ceph
ceph 6105 16.0 11.6 6576336 5694556 ? Ssl mars29 1782:52 /usr/bin/ceph-osd -f --cluster ceph --id 150 --setuser ceph --setgroup ceph
daevel-ob@ssdr712h:~$ free -m
total used free shared buff/cache available
Mem: 47771 45210 1643 17 917 43556
Swap: 0 0 0
# ceph daemon osd.147 config show | grep memory_target
"osd_memory_target": "4294967296",
And there is no recovery / backfilling, the cluster is fine :
$ ceph status
cluster:
id: de035250-323d-4cf6-8c4b-cf0faf6296b1
health: HEALTH_OK
services:
mon: 5 daemons, quorum tolriq,tsyne,olkas,lorunde,amphel
mgr: tsyne(active), standbys: olkas, tolriq, lorunde, amphel
osd: 120 osds: 116 up, 116 in
data:
pools: 20 pools, 12736 pgs
objects: 15.29M objects, 31.1TiB
usage: 101TiB used, 75.3TiB / 177TiB avail
pgs: 12732 active+clean
4 active+clean+scrubbing+deep
io:
client: 72.3MiB/s rd, 26.8MiB/s wr, 2.30kop/s rd, 1.29kop/s wr
On an other host, in the same pool, I see also high memory usage :
daevel-ob@ssdr712g:~$ ps auxw | grep ceph-osd
ceph 6287 6.6 10.6 6027388 5190032 ? Ssl mars21 1511:07 /usr/bin/ceph-osd -f --cluster ceph --id 131 --setuser ceph --setgroup ceph
ceph 6759 7.3 11.2 6299140 5484412 ? Ssl mars21 1665:22 /usr/bin/ceph-osd -f --cluster ceph --id 132 --setuser ceph --setgroup ceph
ceph 7114 7.0 11.7 6576168 5756236 ? Ssl mars21 1612:09 /usr/bin/ceph-osd -f --cluster ceph --id 133 --setuser ceph --setgroup ceph
ceph 7467 7.4 11.1 6244668 5430512 ? Ssl mars21 1704:06 /usr/bin/ceph-osd -f --cluster ceph --id 134 --setuser ceph --setgroup ceph
ceph 7821 7.7 11.1 6309456 5469376 ? Ssl mars21 1754:35 /usr/bin/ceph-osd -f --cluster ceph --id 135 --setuser ceph --setgroup ceph
ceph 8174 6.9 11.6 6545224 5705412 ? Ssl mars21 1590:31 /usr/bin/ceph-osd -f --cluster ceph --id 136 --setuser ceph --setgroup ceph
ceph 8746 6.6 11.1 6290004 5477204 ? Ssl mars21 1511:11 /usr/bin/ceph-osd -f --cluster ceph --id 137 --setuser ceph --setgroup ceph
ceph 9100 7.7 11.6 6552080 5713560 ? Ssl mars21 1757:22 /usr/bin/ceph-osd -f --cluster ceph --id 138 --setuser ceph --setgroup ceph
But ! On a similar host, in a different pool, the problem is less visible :
daevel-ob@ssdr712i:~$ ps auxw | grep ceph-osd
ceph 3617 2.8 9.9 5660308 4847444 ? Ssl mars29 313:05 /usr/bin/ceph-osd -f --cluster ceph --id 151 --setuser ceph --setgroup ceph
ceph 3958 2.3 9.8 5661936 4834320 ? Ssl mars29 256:55 /usr/bin/ceph-osd -f --cluster ceph --id 152 --setuser ceph --setgroup ceph
ceph 4299 2.3 9.8 5620616 4807248 ? Ssl mars29 266:26 /usr/bin/ceph-osd -f --cluster ceph --id 153 --setuser ceph --setgroup ceph
ceph 4643 2.3 9.6 5527724 4713572 ? Ssl mars29 262:50 /usr/bin/ceph-osd -f --cluster ceph --id 154 --setuser ceph --setgroup ceph
ceph 5016 2.2 9.7 5597504 4783412 ? Ssl mars29 248:37 /usr/bin/ceph-osd -f --cluster ceph --id 155 --setuser ceph --setgroup ceph
ceph 5380 2.8 9.9 5700204 4886432 ? Ssl mars29 321:05 /usr/bin/ceph-osd -f --cluster ceph --id 156 --setuser ceph --setgroup ceph
ceph 5724 3.1 10.1 5767456 4953484 ? Ssl mars29 352:55 /usr/bin/ceph-osd -f --cluster ceph --id 157 --setuser ceph --setgroup ceph
ceph 6070 2.7 9.9 5683092 4868632 ? Ssl mars29 309:10 /usr/bin/ceph-osd -f --cluster ceph --id 158 --setuser ceph --setgroup ceph
Is there some memory leak ? Or should I expect that osd_memory_target
(the default 4GB here) is not really followed, and so reduce it ?
Thanks,
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
