On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven <pl@xxxxxxx> wrote: > > Am 24.06.22 um 16:13 schrieb Peter Lieven: > > Am 23.06.22 um 12:59 schrieb Ilya Dryomov: > >> On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven <pl@xxxxxxx> wrote: > >>> Am 22.06.22 um 15:46 schrieb Josh Baergen: > >>>> Hey Peter, > >>>> > >>>>> I found relatively large allocations in the qemu smaps and checked the contents. It contained several hundred repetitions of osd and pool names. We use the default builds on Ubuntu 20.04. Is there a special memory allocator in place that might not clean up properly? > >>>> I'm sure you would have noticed this and mentioned it if it was so - > >>>> any chance the contents of these regions look like log messages of > >>>> some kind? I recently tracked down a high client memory usage that > >>>> looked like a leak that turned out to be a broken config option > >>>> resulting in higher in-memory log retention: > >>>> https://tracker.ceph.com/issues/56093. AFAICT it affects Nautilus+. > >>> Hi Josh, hi Ilya, > >>> > >>> > >>> it seems we were in fact facing 2 leaks with 14.x. Our long running VMs with librbd 14.x have several million items in the osdmap mempool. > >>> > >>> In our testing environment with 15.x I see no unlimited increase in the osdmap mempool (compared this to a second dev host with 14.x client where I see the increase wiht my tests), > >>> > >>> but I still see leaking memory when I generate a lot of osdmap changes, but this in fact seem to be log messages - thanks Josh. > >>> > >>> > >>> So I would appreciate if #56093 would be backported to Octopus before its final release. > >> I picked up Josh's PR that was sitting there unnoticed but I'm not sure > >> it is the issue you are hitting. I think Josh's change just resurrects > >> the behavior where clients stored only up to 500 log entries instead of > >> up to 10000 (the default for daemons). There is no memory leak there, > >> just a difference in how much memory is legitimately consumed. The > >> usage is bounded either way. > >> > >> However in your case, the usage is slowly but constantly growing. > >> In the original post you said that it was observed both on 14.2.22 and > >> 15.2.16. Are you saying that you are no longer seeing it in 15.x? > > > > After I understood whats the background of Josh issue I can confirm that I still see increasing memory which is not caused > > > > by osdmap items and also not by log entries. There must be something else going on. > > > I still see increased memory (heap) usage. Might it be that it is just heap fragmentation? Hi Peter, It could be but you never quantified the issue. What is the actual heap usage you are seeing, how fast is it growing? Is it specific to some particular VMs or does it affect the entire fleet? > > We mainly see data from inside the VM in these memory areas (this might be data from buffered writes), but also librbd data. > > Is it possible that data from buffered writes is not always freed properly? If it's a bug, anything is possible ;) More seriously, I think we need to start from scratch. Initially you suspected the osdmap handling code based on the contents of some QEMU process mappings dumped out with gdb, but now it's VM data? Are osdmap pieces no longer there? What librbd version are you testing on? Thanks, Ilya _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |