Did you check the mempools?
ceph daemon osd.X dump_mempools
This will tell you how much memory is consumed by different components of
the OSD.
Finger in the air, your ram might be consumed by the pg_log.
If osd_pglog from the dump_mempools output is big, then you can lower the
values of the related configuration options:
osd_min_pg_log_entries = 100 (default is 250)
osd_max_pg_log_entries = 500 (default is 10000)
osd_target_pg_log_entries_per_osd = 30000 (default is 300000)
Those are just examples. You can adjust it based on your current memory
consumption and the available amount of RAM.
On Fri, Jun 10, 2022 at 1:21 PM Eugen Block <eblock@xxxxxx> wrote:
> Hi,
>
> could you share more cluster details and what your workload is?
>
> ceph -s
> ceph osd df tree
> ceph orch ls
> ceph osd pools ls detail
>
> How big are your PGs?
>
> Zitat von Mara Sophie Grosch <littlefox@xxxxxxxxxx>:
>
> > Hi,
> >
> > good catch with the way too low memory target, I wanted to configure 1
> > GiB not 1 MiB. I'm aware it's low, but removed anyway for testing - it
> > sadly didn't change anything.
> >
> > I customize the config mostly for dealing problems I have, something in
> > my setup makes the OSDs eat lots of memory in normal operation, just
> > gradually increasing .. I'd send metrics if my monitoring was up again
> > ^^' That maybe being some form of cache was the reason for that config
> > line (it does not seem to be cache).
> >
> > I have found a way to deal with my cluster, using ceph-objectstore-tool
> > to export-remove all PGs from the OSDs, getting them online and happy
> > again and then importing a few PGs at a time in one OSD and let it
> > backfill to the others.
> >
> > The problem of eating very much memory on startup manifests with some
of
> > the PGs only, but for those it goes up to ~50GiB. Problematic: when it
> > has multiple of those PGs in its storage, it's handling those in
> > parallel - needing even more memory of course, until it finally gets
> > OOM-killed.
> >
> > So.. seems I can get my cluster running again, only limited by my
> > internet upload now. Any hints why it eats a lot of memory in normal
> > operation would still be appreciated.
> >
> > Best, Mara
> >
> >
> > Am Wed, Jun 08, 2022 at 09:05:52AM +0000 schrieb Eugen Block:
> >> It's even worse, you only give them 1MB, not GB.
> >>
> >> Zitat von Eugen Block <eblock@xxxxxx>:
> >>
> >>> Hi,
> >>>
> >>> is there any reason you use custom configs? Most of the defaults
> >>> work well. But you only give your OSDs 1 GB of memory, that is way
> >>> too low except for an idle cluster without much data. I recommend
> >>> to remove the line
> >>>
> >>> osd_memory_target = 1048576
> >>>
> >>> and let ceph handle it. I didn't install Quincy yet but in a
> >>> healthy cluster the OSDs will take around 3 GB of memory, maybe 4,
> >>> so you should be good with your setup.
> >>>
> >>> Regards,
> >>> Eugen
> >>>
> >>> Zitat von Mara Sophie Grosch <littlefox@xxxxxxxxxx>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I have a currently-down ceph cluster
> >>>> * v17.2.0 / quay.io/v17.2.0-20220420
> >>>> * 3 nodes, 4 OSDs
> >>>> * around 1TiB used/3TiB total
> >>>> * probably enough resources
> >>>> - two of those nodes have 64GiB memory, the third has 16GiB
> >>>> - one of the 64GiB nodes runs two OSDs, as it's a physical node with
> >>>> 2 NVMe drives
> >>>> * provisioned via Rook and running in my Kubernetes cluster
> >>>>
> >>>> After some upgrades yesterday (system packages on the nodes) and
today
> >>>> (Kubernetes to latest version), I wanted to reboot my nodes. The
drain
> >>>> of the first node put a lot of stress on the other OSDs, making them
> go
> >>>> OOM - but I think that probably is a bug already, as at least one of
> >>>> those nodes has enough resources (64GiB memory, physical machine,
> ~40GiB
> >>>> surely free - but don't have metrics rn as everything is down).
> >>>>
> >>>> I'm now seeing all OSDs going into OOM right on startup, from what
it
> >>>> looks like everything is fine until right after `load_pgs` - as soon
> as
> >>>> it activates some PGs, memory usage increases _a lot_ (from ~4-5GiB
> >>>> RES before to .. 60GiB, though that depends on the free memory on
the
> >>>> node).
> >>>>
> >>>> Because of this, I cannot get any of them online again and need
advice
> >>>> what to do and what info might be useful. Logs of one of those OSDs
> are
> >>>> here[1] (captured via kubectl logs, so something right from start
> might
> >>>> be missing - happy to dig deeper if you need more) and my changed
> >>>> ceph.conf entries are here[2]. I had `bluefs_buffered_io = false`
> until
> >>>> today, changed it to true based on a suggestion in another debug
> >>>> thread[3]
> >>>>
> >>>> Any hint is greatly appreciated, many thanks
> >>>> Mara Grosch
> >>>>
> >>>> [1] https://pastebin.com/VFczNqUk
> >>>> [2] https://pastebin.com/QXust5XD
> >>>> [3]
> >>>>
>
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/CBPXLPWEVZLZE55WAQSMB7KSIQPV5I76/
> >>
> >>
> >>
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
