It's also possible you're running into large pglog entries - any chance you're running RGW and there's an s3:CopyObject workload hitting an object that was uploaded with MPU? https://tracker.ceph.com/issues/56707 If that's the case, you can inject a much smaller value for osd_min_pg_log_entries and osd_max_pg_log_entries (ceph tell osd.* config set osd_min_pg_log_entries 500 - repeat for max) to relieve memory pressure. Josh On Tue, Nov 29, 2022 at 3:10 PM Frank Schilder <frans@xxxxxx> wrote: > > Hi, it sounds like you might be affected by the pg_log dup bug: > > # Check if any OSDs are affected by the pg dup problem > sudo -i ceph tell "osd.*" perf dump | grep -e pglog -e "osd\\." > > If any osd_pglog_items>>1M check https://www.clyso.com/blog/osds-with-unlimited-ram-growth/ > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Gregory Farnum <gfarnum@xxxxxxxxxx> > Sent: 29 November 2022 22:25:54 > To: Joshua Timmer > Cc: ceph-users@xxxxxxx > Subject: Re: Implications of pglog_hardlimit > > On Tue, Nov 29, 2022 at 1:18 PM Joshua Timmer <mrjoshuatimmer@xxxxxxxxx> > wrote: > > > I've got a cluster in a precarious state because several nodes have run > > out of memory due to extremely large pg logs on the osds. I came across > > the pglog_hardlimit flag which sounds like the solution to the issue, > > but I'm concerned that enabling it will immediately truncate the pg logs > > and possibly drop some information needed to recover the pgs. There are > > many in degraded and undersized states right now as nodes are down. Is > > it safe to enable the flag in this state? The cluster is running > > luminous 12.2.13 right now. > > > The hard limit will truncate the log, but all the data goes into the > backing bluestore/filestore instance at the same time. The pglogs are used > for two things: > 1) detecting replayed client operations and sending the same answer back on > replays, so shorter logs means a shorter time window of detection but > shouldn’t be an issue; > 2) enabling log-based recovery of pgs where OSDs with overlapping logs can > identify exactly which objects have been modified and only moving them. > > So if you set the hard limit, it’s possible you’ll induce more backfill as > fewer logs overlap. But no data will be lost. > -Greg > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |