On Tue, Jul 26, 2022 at 1:41 PM Peter Lieven <pl@xxxxxxx> wrote: > > Am 21.07.22 um 17:50 schrieb Ilya Dryomov: > > On Thu, Jul 21, 2022 at 11:42 AM Peter Lieven <pl@xxxxxxx> wrote: > >> Am 19.07.22 um 17:57 schrieb Ilya Dryomov: > >>> On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven <pl@xxxxxxx> wrote: > >>>> Am 24.06.22 um 16:13 schrieb Peter Lieven: > >>>>> Am 23.06.22 um 12:59 schrieb Ilya Dryomov: > >>>>>> On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven <pl@xxxxxxx> wrote: > >>>>>>> Am 22.06.22 um 15:46 schrieb Josh Baergen: > >>>>>>>> Hey Peter, > >>>>>>>> > >>>>>>>>> I found relatively large allocations in the qemu smaps and checked the contents. It contained several hundred repetitions of osd and pool names. We use the default builds on Ubuntu 20.04. Is there a special memory allocator in place that might not clean up properly? > >>>>>>>> I'm sure you would have noticed this and mentioned it if it was so - > >>>>>>>> any chance the contents of these regions look like log messages of > >>>>>>>> some kind? I recently tracked down a high client memory usage that > >>>>>>>> looked like a leak that turned out to be a broken config option > >>>>>>>> resulting in higher in-memory log retention: > >>>>>>>> https://tracker.ceph.com/issues/56093. AFAICT it affects Nautilus+. > >>>>>>> Hi Josh, hi Ilya, > >>>>>>> > >>>>>>> > >>>>>>> it seems we were in fact facing 2 leaks with 14.x. Our long running VMs with librbd 14.x have several million items in the osdmap mempool. > >>>>>>> > >>>>>>> In our testing environment with 15.x I see no unlimited increase in the osdmap mempool (compared this to a second dev host with 14.x client where I see the increase wiht my tests), > >>>>>>> > >>>>>>> but I still see leaking memory when I generate a lot of osdmap changes, but this in fact seem to be log messages - thanks Josh. > >>>>>>> > >>>>>>> > >>>>>>> So I would appreciate if #56093 would be backported to Octopus before its final release. > >>>>>> I picked up Josh's PR that was sitting there unnoticed but I'm not sure > >>>>>> it is the issue you are hitting. I think Josh's change just resurrects > >>>>>> the behavior where clients stored only up to 500 log entries instead of > >>>>>> up to 10000 (the default for daemons). There is no memory leak there, > >>>>>> just a difference in how much memory is legitimately consumed. The > >>>>>> usage is bounded either way. > >>>>>> > >>>>>> However in your case, the usage is slowly but constantly growing. > >>>>>> In the original post you said that it was observed both on 14.2.22 and > >>>>>> 15.2.16. Are you saying that you are no longer seeing it in 15.x? > >>>>> After I understood whats the background of Josh issue I can confirm that I still see increasing memory which is not caused > >>>>> > >>>>> by osdmap items and also not by log entries. There must be something else going on. > >>>> I still see increased memory (heap) usage. Might it be that it is just heap fragmentation? > >>> Hi Peter, > >>> > >>> It could be but you never quantified the issue. What is the actual > >>> heap usage you are seeing, how fast is it growing? Is it specific to > >>> some particular VMs or does it affect the entire fleet? > >> > >> Hi Ilya, > >> > >> > >> I see the issue across the fleet. The memory increases about 200KB/day per attached drive. > >> > >> Same hypervisor with attached iSCSI storage - no issue. > >> > >> > >> However, the memory that is increasing is not listed as heap under /proc/{pid}/smaps. > >> > >> Does librbd use its own memory allocator? > > Hi Peter, > > > > By default, librbd uses tcmalloc. > > > >> > >> I am still testing with 15.x as I mainly have long running VMs in our production environment. > >> > >> With 14.x we had an additional issue with the ospmaps not beeing freed. That is gone with 15.x > >> > >> > >> I will try with a patched qemu that allocated the write buffers inside qemu and set disable_zero_copy_write = true. > >> > >> to see if this makes any difference. > > We are unlikely to be able to do anything about 15.x at this point so > > I'd encourage you to try 17.x. That said, any new information would be > > helpful. > > > Update from my side. Even with Quincy I see guest data in librbd memory even if disable_zero_copy_write is set to false. > > How can that happen? Any ideas? rbd_cache is off. Hi Peter, I don't think the read path is zero-copy. This limitation comes from librados, not from librbd. See https://github.com/ceph/ceph/pull/26696. Thanks, Ilya _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |