I think you may have hit the same bug we encountered. Cory submitted a fix, see if it fits what you've encountered: https://github.com/ceph/ceph/pull/46727 (backport to Pacific here: https://github.com/ceph/ceph/pull/46877 ) https://tracker.ceph.com/issues/54172 On Fri, Jul 15, 2022 at 8:52 AM Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx> wrote: > We have two clusters one 14.2.22 -> 16.2.7 -> 16.2.9 > > Another 16.2.7 -> 16.2.9 > > Both with a multi disk (spinner block / ssd block.db) and both CephFS > around 600 OSDs each with combo of rep-3 and 8+3 EC data pools. Examples of > stuck scrubbing PGs from all of the pools. > > They have generally been behind on scrubbing which we attributed to simply > being large disks (10TB) with a heavy write load and the OSDs just having > trouble keeping up. On closer inspection it appears we have many PGs that > have been lodged in a deep scrubbing state on one cluster for 2 weeks and > another for 7 weeks. Wondering if others have been experiencing anything > similar. The only example of PGs being stuck scrubbing I have seen in the > past has been related to snaptrim PG state but we arent doing anything with > snapshots in these new clusters. > > Granted my cluster has been warning me with "pgs not deep-scrubbed in time" > and its on me for not looking more closely into why. Perhaps a separate > warning of "PG Stuck Scrubbing for greater than 24 hours" or similar might > be helpful to an operator. > > In any case I was able to get scrubs proceeding again by restarting the > primary OSD daemon in the PGs which were stuck. Will monitor closely for > additional stuck scrubs. > > > Respectfully, > > *Wes Dillingham* > wes@xxxxxxxxxxxxxxxxx > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |