Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven <pl@xxxxxxx> wrote:
Am 22.06.22 um 15:46 schrieb Josh Baergen:
Hey Peter,
I found relatively large allocations in the qemu smaps and checked the contents. It contained several hundred repetitions of osd and pool names. We use the default builds on Ubuntu 20.04. Is there a special memory allocator in place that might not clean up properly?
I'm sure you would have noticed this and mentioned it if it was so -
any chance the contents of these regions look like log messages of
some kind? I recently tracked down a high client memory usage that
looked like a leak that turned out to be a broken config option
resulting in higher in-memory log retention:
https://tracker.ceph.com/issues/56093. AFAICT it affects Nautilus+.
Hi Josh, hi Ilya,
it seems we were in fact facing 2 leaks with 14.x. Our long running VMs with librbd 14.x have several million items in the osdmap mempool.
In our testing environment with 15.x I see no unlimited increase in the osdmap mempool (compared this to a second dev host with 14.x client where I see the increase wiht my tests),
but I still see leaking memory when I generate a lot of osdmap changes, but this in fact seem to be log messages - thanks Josh.
So I would appreciate if #56093 would be backported to Octopus before its final release.
I picked up Josh's PR that was sitting there unnoticed but I'm not sure
it is the issue you are hitting. I think Josh's change just resurrects
the behavior where clients stored only up to 500 log entries instead of
up to 10000 (the default for daemons). There is no memory leak there,
just a difference in how much memory is legitimately consumed. The
usage is bounded either way.
Hi Ilya,
in his initial message Josh wrote that he was debugging a high client memory usage when he hit issue #56093 so I was thinking he discovered something that was looking like a leak.
However in your case, the usage is slowly but constantly growing.
In the original post you said that it was observed both on 14.2.22 and
15.2.16. Are you saying that you are no longer seeing it in 15.x?
The osdmap buffer pool usage is not an issue anymore. However, even with 15.x Qemu's PSS usage is still constantly increasing. Can't say if there is an upper bound.
After 1 day of running I see an ~11MB allocation that belongs to librbd (librados) and that contains log messages (over 10k in my case) and other stuff. And that allocation is increasing over time.
VmFlags: mr mw me sd
7f8ce71ca000-7f8ce7d2b000 rw-p 00000000 00:00 0
Size: 11652 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 10652 kB
Pss: 10652 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 10652 kB
Referenced: 10652 kB
Anonymous: 10652 kB
The mempool for this librbd instance shows the following info:
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 0,
"bytes": 0
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 0,
"bytes": 0
},
"bluestore_cache_meta": {
"items": 0,
"bytes": 0
},
"bluestore_cache_other": {
"items": 0,
"bytes": 0
},
"bluestore_Buffer": {
"items": 0,
"bytes": 0
},
"bluestore_Extent": {
"items": 0,
"bytes": 0
},
"bluestore_Blob": {
"items": 0,
"bytes": 0
},
"bluestore_SharedBlob": {
"items": 0,
"bytes": 0
},
"bluestore_inline_bl": {
"items": 0,
"bytes": 0
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 0,
"bytes": 0
},
"bluestore_writing_deferred": {
"items": 0,
"bytes": 0
},
"bluestore_writing": {
"items": 0,
"bytes": 0
},
"bluefs": {
"items": 0,
"bytes": 0
},
"bluefs_file_reader": {
"items": 0,
"bytes": 0
},
"bluefs_file_writer": {
"items": 0,
"bytes": 0
},
"buffer_anon": {
"items": 124,
"bytes": 211847
},
"buffer_meta": {
"items": 0,
"bytes": 0
},
"osd": {
"items": 0,
"bytes": 0
},
"osd_mapbl": {
"items": 0,
"bytes": 0
},
"osd_pglog": {
"items": 0,
"bytes": 0
},
"osdmap": {
"items": 684,
"bytes": 25064
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 808,
"bytes": 236911
}
}
}
Best
Peter
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
