A problem with the CEPH PG state getting stuck

"=?gb18030?b?y9Wy7Ln+tvuy0w==?=" <2644294460@xxxxxx> · Tue, 23 Jul 2024 10:48:52 +0800

My ceph cluster sometimes gets stuck in the active+clean+snaptrim state when doing regular snapshot deletion, and the corresponding pg does not change for a long time. As follows:

27.7c&nbsp; &nbsp; &nbsp;14350&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; 38073876582&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp;2864&nbsp; &nbsp; &nbsp; 3000&nbsp; &nbsp; &nbsp; &nbsp; active+clean+snaptrim&nbsp; &nbsp; &nbsp;9h&nbsp; 43777'677286&nbsp; 43777:1018248&nbsp; &nbsp; [52,55,20,36,29,91,9,63,14,2]p52&nbsp; &nbsp; [52,55,20,36,29,91,9,63,14,2]p52&nbsp; 2024-07-21T13:57:12.602903+0000&nbsp; 2024-07-20T11:02:06.953985+0000&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;52&nbsp; queued for scrub&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
27.c1&nbsp; &nbsp; &nbsp;14055&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; 37408096205&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp;2754&nbsp; &nbsp; &nbsp; 3000&nbsp; &nbsp; &nbsp; &nbsp; active+clean+snaptrim&nbsp; &nbsp; &nbsp;9h&nbsp; 43777'676887&nbsp; &nbsp;43777:952875&nbsp; &nbsp; &nbsp;[0,21,39,62,26,58,41,86,66,2]p0&nbsp; &nbsp; &nbsp;[0,21,39,62,26,58,41,86,66,2]p0&nbsp; 2024-07-22T02:26:43.470883+0000&nbsp; 2024-07-20T05:09:23.763918+0000&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;44&nbsp; periodic scrub scheduled @ 2024-07-23T07:46:50.084316+0000&nbsp; &nbsp; &nbsp;
27.19b&nbsp; &nbsp; 14711&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; 38926389125&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp;2765&nbsp; &nbsp; &nbsp; 3000&nbsp; &nbsp; &nbsp; &nbsp; active+clean+snaptrim&nbsp; &nbsp; &nbsp;9h&nbsp; 43777'698478&nbsp; &nbsp;43777:849574&nbsp; &nbsp; &nbsp; [5,7,29,14,61,10,22,0,19,37]p5&nbsp; &nbsp; &nbsp; [5,7,29,14,61,10,22,0,19,37]p5&nbsp; 2024-07-22T03:28:39.036189+0000&nbsp; 2024-07-17T14:00:43.082766+0000&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;40&nbsp; periodic scrub scheduled @ 2024-07-23T12:30:57.488823+0000&nbsp; &nbsp; &nbsp;
27.1a3&nbsp; &nbsp; 14323&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; 37918899397&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp;2930&nbsp; &nbsp; &nbsp; 3000&nbsp; &nbsp; &nbsp; &nbsp; active+clean+snaptrim&nbsp; &nbsp; &nbsp;9h&nbsp; 43777'675713&nbsp; &nbsp;43777:943324&nbsp; &nbsp;[52,4,10,46,20,69,49,39,44,30]p52&nbsp; &nbsp;[52,4,10,46,20,69,49,39,44,30]p52&nbsp; 2024-07-21T11:15:16.442506+0000&nbsp; 2024-07-20T04:10:45.391550+0000&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;50&nbsp; queued for scrub&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;

No change for 9 hours. Then, after I typed the command "ceph pg repeer 27.7c", the corresponding pg state was restored to normal. I don't know what this "repeer" command is for, will it have any effect on pg? What's the difference between "repeer" and "ceph pg repair"? Then, why does the pg get stuck so often? Thanks for your help!

My ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx