Hi, Igor.
Thank you for your reply!
ceph tell osd.10 status| grep map; echo ---; sleep 600; ceph tell osd.10
status| grep map
"oldest_map": 2408326,
"newest_map": 2635631,
---
"oldest_map": 2408326,
"newest_map": 2635647,
ceph version is 16.2.13
ceph health:
HEALTH_WARN mons a,b,c,d,e are using a lot of disk space; 8 backfillfull
osd(s); 143 nearfull osd(s); Low space hindering backfill (add storage if
this doesn't resolve itself): 25 pgs backfill_toofull; (muted:
BLUESTORE_NO_PER_POOL_OMAP PG_NOT_DEEP_SCRUBBED POOL_BACKFILLFULL
POOL_NEARFULL)
(MUTED, STICKY) [WRN] BLUESTORE_NO_PER_POOL_OMAP: 1756 OSD(s) reporting
legacy (not per-pool) BlueStore omap usage stats
osd.1 legacy (not per-pool) omap detected, suggest to run store repair
to benefit from per-pool omap usage statistics
...
osd.2772 legacy (not per-pool) omap detected, suggest to run store
repair to benefit from per-pool omap usage statistics
[WRN] MON_DISK_BIG: mons a,b,c,d,e are using a lot of disk space
mon.a is 56 GiB >= mon_data_size_warn (15 GiB)
...
[WRN] OSD_BACKFILLFULL: 8 backfillfull osd(s)
osd.12 is backfill full
...
[WRN] OSD_NEARFULL: 143 nearfull osd(s)
osd.1 is near full
...
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this
doesn't resolve itself): 25 pgs backfill_toofull
pg 10.6ea is active+remapped+backfill_toofull, acting [1507,941,2649]
...
(MUTED, STICKY) [WRN] PG_NOT_DEEP_SCRUBBED: 2302 pgs not deep-scrubbed in
time
pg 10.7ffe not deep-scrubbed since 2024-08-23T19:28:22.749150+0300
...
(MUTED, STICKY) [WRN] POOL_BACKFILLFULL: 19 pool(s) backfillfull
pool '.rgw.root' is backfillfull
pool '.rgw.control' is backfillfull
pool '.rgw' is backfillfull
pool '.rgw.gc' is backfillfull
pool '.users.uid' is backfillfull
pool '.users' is backfillfull
pool '.usage' is backfillfull
pool '.intent-log' is backfillfull
pool '.log' is backfillfull
pool '.rgw.buckets' is backfillfull
pool '.rgw.buckets.extra' is backfillfull
pool '.rgw.buckets.index' is backfillfull
pool '.users.email' is backfillfull
pool 'fs1_meta' is backfillfull
pool 'fs1_data' is backfillfull
pool 'fs1_tmp' is backfillfull
pool 'device_health_metrics' is backfillfull
pool 'default.rgw.meta' is backfillfull
ср, 18 сент. 2024 г. в 18:23, Igor Fedotov <igor.fedotov@xxxxxxxx>:
> Hi Alexander,
>
> I recall a couple of cases when permanent osdmap epoch growth has been
> filling OSD with relevant osd map info. Which could be tricky to catch.
>
> Please run 'ceph tell osd.N status" for a couple of affected OSDs twice
> within e.g. 10 min interval.
>
> Then check the delta between oldest_map and newest_map fields - neither
> the delta should be very large (hundreds of thousands) nor it should grow
> rapidly within the observed interval.
> If so - please share these reports, 'ceph health detailed' output and
> exact Ceph release version you're using.
>
>
> Thanks,
>
> Igor
>
>
> On 9/18/2024 2:32 PM, Александр Руденко wrote:
>
> erstand, the majority of these pools contain only
>
> --
> Igor Fedotov
> Ceph Lead Developer
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
