I halfway suspect that something akin to the speculation in https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/7MWAHAY7NCJK2DHEGO6MO4SWTLPTXQMD/ is going on. Below are reservations reported by a random OSD that serves (mostly) an EC RGW bucket pool. This is with the mclock override on and the usual three backfill/recovery tunables set to 7 (bumped to get more OSDs backfilling after I changed to rack failure domain, having 50+ % of objects remapped makes me nervous and I want convergence. 3 happens to be the value of osd_recovery_max_active_hdd , so maybe there is some interaction between EC and how osd_recovery_max_active is derived and used? Complete wild-ass speculation. Just for grins, after `ceph osd down 313` * local_reservations incremented * remote_reservations decreased somewhat * cluster aggregate recovery speed increased for at least the short term [root@rook-ceph-osd-313-6f84bc5bd5-hr825 ceph]# ceph daemon osd.313 dump_recovery_reservations { "local_reservations": { "max_allowed": 7, "min_priority": 0, "queues": [], "in_progress": [ { "item": "21.161es0", "prio": 110, "can_preempt": true }, { "item": "21.180bs0", "prio": 110, "can_preempt": true }, { "item": "21.1e0as0", "prio": 110, "can_preempt": true } ] }, "remote_reservations": { "max_allowed": 7, "min_priority": 0, "queues": [ { "priority": 110, "items": [ { "item": "21.1d18s5", "prio": 110, "can_preempt": true }, { "item": "21.7d0s2", "prio": 110, "can_preempt": true }, { "item": "21.766s5", "prio": 110, "can_preempt": true }, { "item": "21.373s1", "prio": 110, "can_preempt": true }, { "item": "21.1a8es1", "prio": 110, "can_preempt": true }, { "item": "21.2das2", "prio": 110, "can_preempt": true }, { "item": "21.14a0s2", "prio": 110, "can_preempt": true }, { "item": "21.c7fs5", "prio": 110, "can_preempt": true }, { "item": "21.18e5s5", "prio": 110, "can_preempt": true }, { "item": "21.54ds2", "prio": 110, "can_preempt": true }, { "item": "21.79bs4", "prio": 110, "can_preempt": true }, { "item": "21.15c3s2", "prio": 110, "can_preempt": true }, { "item": "21.e15s4", "prio": 110, "can_preempt": true }, { "item": "21.226s3", "prio": 110, "can_preempt": true }, { "item": "21.adfs2", "prio": 110, "can_preempt": true }, { "item": "21.184bs4", "prio": 110, "can_preempt": true }, { "item": "21.f43s3", "prio": 110, "can_preempt": true }, { "item": "21.f5cs4", "prio": 110, "can_preempt": true }, { "item": "21.1300s3", "prio": 110, "can_preempt": true }, { "item": "21.11d8s3", "prio": 110, "can_preempt": true }, { "item": "21.5cfs5", "prio": 110, "can_preempt": true }, { "item": "21.1342s2", "prio": 110, "can_preempt": true }, { "item": "21.1846s5", "prio": 110, "can_preempt": true }, { "item": "21.c60s2", "prio": 110, "can_preempt": true }, { "item": "21.f1cs3", "prio": 110, "can_preempt": true }, { "item": "21.314s5", "prio": 110, "can_preempt": true }, { "item": "21.1be3s5", "prio": 110, "can_preempt": true } ] } ], "in_progress": [ { "item": "21.2fds0", "prio": 110, "can_preempt": true }, { "item": "21.4e6s1", "prio": 110, "can_preempt": true }, { "item": "21.c4cs3", "prio": 110, "can_preempt": true }, { "item": "21.f7as1", "prio": 110, "can_preempt": true }, { "item": "21.103es3", "prio": 110, "can_preempt": true }, { "item": "21.1d09s1", "prio": 110, "can_preempt": true }, { "item": "21.1f53s3", "prio": 110, "can_preempt": true } ] } } > On May 2, 2024, at 10:02, Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx> wrote: > > In our case it was with a EC pool as well. I believe the PG state was degraded+recovering / recovery_wait and iirc the PGs just simply sat in the recovering state without any progress (degraded PG object count did not decline). A repeer of the PG was attempted but no success there. A restart of all the OSDs for the given PGs was attempted under mclock. That didnt work. Switching to wpq for all OSDS in the given PG did resolve the issue. This was on a 17.2.7 cluster. > > Respectfully, > > Wes Dillingham > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > wes@xxxxxxxxxxxxxxxxx <mailto:wes@xxxxxxxxxxxxxxxxx> > > > > > On Thu, May 2, 2024 at 9:54 AM Sridhar Seshasayee <sseshasa@xxxxxxxxxx <mailto:sseshasa@xxxxxxxxxx>> wrote: >> > >> > Multiple people -- including me -- have also observed backfill/recovery >> > stop completely for no apparent reason. >> > >> > In some cases poking the lead OSD for a PG with `ceph osd down` restores, >> > in other cases it doesn't. >> > >> > Anecdotally this *may* only happen for EC pools on HDDs but that sample >> > size is small. >> > >> > >> Thanks for the information. We will try and reproduce this locally with EC >> pools and investigate this further. >> I will revert with a tracker for this. >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |