Hi Everyone,
My Nautilus cluster of 6 nodes, 180 OSDs, is having a weird issue I don’t know how to troubleshoot.
I started receiving health warning issues, and the number of PGS not deep-scrubbed in time has been increasing.
# ceph health detail
HEALTH_WARN 3013 pgs not scrubbed in time
PG_NOT_SCRUBBED 3013 pgs not scrubbed in time
pg 1.e99 not scrubbed since 2022-12-21 11:01:10.403525
pg 1.e94 not scrubbed since 2022-12-18 06:26:14.086410
pg 3.e91 not scrubbed since 2022-12-17 03:00:25.104174
pg 1.e90 not scrubbed since 2022-12-18 03:31:44.747218
pg 1.e8e not scrubbed since 2022-12-21 12:04:17.111762
pg 1.e89 not scrubbed since 2022-12-18 07:20:13.328540
...
2963 more pgs…
Strangely, ceph pg dump gives shows every other PG with 0 objects. An attempt to perform a deep scrub (or scrub) on one of these PGs does nothing. The cluster appears to be running fine, but obviously there’s an issue. What should my next steps be to troubleshoot ?
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN
3.e9b 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 22:49:07.629579 0'0 23686:19820 [28,79] 28 [28,79] 28 0'0 2022-12-31 22:49:07.629508 0'0 2022-12-31 22:49:07.629508 0
1.e99 60594 0 0 0 0 177433523272 0 0 3046 3046 active+clean 2022-12-21 14:35:08.175858 23686'268137 23686:1732399 [178,115] 178 [178,115] 178 23675'267613 2022-12-21 11:01:10.403525 23675'267613 2022-12-21 11:01:10.403525 0
3.e9a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 09:16:48.644619 0'0 23686:22855 [51,140] 51 [51,140] 51 0'0 2022-12-31 09:16:48.644568 0'0 2022-12-30 02:35:23.367344 0
1.e98 59962 0 0 0 0 177218669411 0 0 3035 3035 active+clean 2022-12-28 14:14:49.908560 23686'265576 23686:1357499 [92,86] 92 [92,86] 92 23686'265445 2022-12-28 14:14:49.908522 23686'265445 2022-12-28 14:14:49.908522 0
3.e95 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 06:09:39.442932 0'0 23686:22757 [48,83] 48 [48,83] 48 0'0 2022-12-31 06:09:39.442879 0'0 2022-12-18 09:33:47.892142 0
1.e97 60062 0 0 0 0 176721095235 0 0 3050 3050 active+clean 2022-12-31 21:19:33.758473 23686'267934 23686:1514273 [137,123] 137 [137,123] 137 23686'267916 2022-12-31 21:19:33.758417 23686'267713 2022-12-27 19:16:27.025326 0
3.e94 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 10:00:38.864773 0'0 23686:18478 [101,1] 101 [101,1] 101 0'0 2022-12-31 10:00:38.864730 0'0 2022-12-28 22:28:13.790168 0
1.e96 59753 0 0 0 0 175411602155 0 0 3083 3083 active+clean 2022-12-28 14:13:32.186265 23686'264255 23686:1676359 [54,170] 54 [54,170] 54 23686'264120 2022-12-28 14:13:32.186220 23686'264120 2022-12-28 14:13:32.186220 0
3.e97 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 11:11:13.202546 0'0 23686:7175 [57,112] 57 [57,112] 57 0'0 2023-01-01 11:11:13.202473 0'0 2023-01-01 11:11:13.202473 0
1.e95 60091 0 0 0 0 176012659050 0 0 3033 3033 active+clean 2022-12-31 08:17:27.280405 23686'267681 23686:1441943 [135,133] 135 [135,133] 135 23686'267648 2022-12-31 08:17:27.280354 23686'267555 2022-12-28 15:45:47.863185 0
3.e96 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 10:57:01.640503 0'0 23686:18395 [116,8] 116 [116,8] 116 0'0 2023-01-01 10:57:01.637962 0'0 2023-01-01 10:57:01.637962 0
1.e94 59926 0 0 0 0 176764011646 0 0 3099 3099 active+clean+scrubbing+deep 2023-01-01 11:11:14.074902 23686'265734 23686:1438653 [40,56] 40 [40,56] 40 23662'264884 2022-12-18 06:26:14.086410 23662'264714 2022-12-13 08:04:24.265324 0
3.e91 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-17 03:00:25.104235 0'0 23685:1849 [77,31] 77 [77,31] 77 0'0 2022-12-17 03:00:25.104174 0'0 2022-12-15 21:38:28.004594 0
1.e93 60193 0 0 0 0 177054652617 0 0 3054 3054 active+clean 2023-01-01 02:21:23.871342 23686'268286 23686:1470308 [72,169] 72 [72,169] 72 23686'268274 2023-01-01 02:21:23.871272 23686'268169 2022-12-29 17:21:15.256027 0
3.e90 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 19:35:50.398721 0'0 23686:11647 [125,45] 125 [125,45] 125 0'0 2022-12-31 19:35:50.398635 0'0 2022-12-31 19:35:50.398635 0
1.e92 60342 0 0 0 0 177585802610 0 0 3098 3098 active+clean 2022-12-31 11:20:33.735990 23686'265365 23686:1493691 [47,91] 47 [47,91] 47 23686'265321 2022-12-31 11:20:33.735945 23686'265321 2022-12-31 11:20:33.735945 0
3.e93 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 18:34:30.921757 0'0 23686:19914 [28,101] 28 [28,101] 28 0'0 2022-12-31 18:34:30.921695 0'0 2022-12-20 01:15:26.329060 0
1.e91 59922 0 0 0 0 176178294337 0 0 3078 3078 active+clean 2022-12-31 16:23:13.129135 23686'266109 23686:1498886 [72,157] 72 [72,157] 72 23686'266084 2022-12-31 16:23:13.129091 23686'266084 2022-12-31 16:23:13.129091 0
3.e92 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 03:21:41.819236 0'0 23686:1548 [73,68] 73 [73,68] 73 0'0 2023-01-01 03:21:41.816785 0'0 2022-12-27 03:21:17.768390 0
1.e90 59885 0 0 0 0 176295573088 0 0 3059 3059 active+clean+scrubbing+deep 2023-01-01 11:11:15.428300 23686'264880 23686:1309008 [39,58] 39 [39,58] 39 23662'264095 2022-12-18 03:31:44.747218 23662'264007 2022-12-16 01:42:56.447456 0
3.e8d 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-28 13:45:08.250912 0'0 23686:13114 [10,58] 10 [10,58] 10 0'0 2022-12-28 13:45:08.250851 0'0 2022-12-28 13:45:08.250851 0
1.e8f 59858 0 0 0 0 175774668550 0 0 3013 3013 active+clean 2022-12-28 14:14:45.310609 23686'268346 23686:1246625 [5,9] 5 [5,9] 5 23686'268182 2022-12-28 14:14:45.310564 23686'268182 2022-12-28 14:14:45.310564 0
3.e8c 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 10:49:45.386653 0'0 23686:11814 [150,100] 150 [150,100] 150 0'0 2023-01-01 10:49:45.386577 0'0 2022-12-27 09:43:45.177707 0
1.e8e 59897 0 0 0 0 175613850948 0 0 3053 3053 active+clean+scrubbing+deep 2023-01-01 11:11:21.286209 23686'267818 23686:1778964 [106,120] 106 [106,120] 106 23664'267253 2022-12-21 12:04:17.111762 23662'266968 2022-12-15 00:56:58.941970 0
3.e8f 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-28 13:45:06.708181 0'0 23686:22818 [66,64] 66 [66,64] 66 0'0 2022-12-28 13:45:06.708120 0'0 2022-12-28 13:45:06.708120 0
1.e8d 60464 0 0 0 0 177786454089 0 0 3088 3088 active+clean 2022-12-31 19:37:07.860682 23686'266321 23686:1723741 [82,136] 82 [82,136] 82 23686'266298 2022-12-31 19:37:07.860638 23686'266242 2022-12-30 15:30:59.925650 0
3.e8e 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 10:17:48.873972 0'0 23686:11722 [145,88] 145 [145,88] 145 0'0 2022-12-31 10:17:48.873922 0'0 2022-12-22 19:29:59.342170 0
1.e8c 59747 0 0 0 0 175420690634 0 0 3011 3011 active+clean 2022-12-28 14:14:10.375574 23686'264942 23686:1426381 [167,79] 167 [167,79] 167 23686'264765 2022-12-28 14:14:10.375538 23686'264765 2022-12-28 14:14:10.375538 0
3.e89 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 11:10:25.652472 0'0 23686:12379 [176,76] 176 [176,76] 176 0'0 2022-12-31 11:10:25.652431 0'0 2022-12-31 11:10:25.652431 0
1.e8b 60177 0 0 0 0 176212803132 0 0 3084 3084 active+clean 2022-12-31 14:34:27.266317 23686'266021 23686:1390892 [61,54] 61 [61,54] 61 23686'265981 2022-12-31 14:34:27.266247 23662'265380 2022-12-18 03:30:40.569564 0
3.e88 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 22:12:22.542301 0'0 23686:22513 [84,109] 84 [84,109] 84 0'0 2022-12-31 22:12:22.542239 0'0 2022-12-21 19:52:56.474546 0
1.e8a 59937 0 0 0 0 175967277330 0 0 3092 3092 active+clean 2023-01-01 06:30:33.023442 23686'266318 23686:1569176 [73,117] 73 [73,117] 73 23686'266314 2023-01-01 06:30:33.023360 23686'266182 2022-12-28 10:24:38.743599 0
3.e8b 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 09:42:09.452649 0'0 23686:11736 [123,135] 123 [123,135] 123 0'0 2023-01-01 09:42:09.452584 0'0 2022-12-22 21:10:31.693983 0
1.e89 60399 0 0 0 0 177399669362 0 0 3080 3080 active+clean 2022-12-18 07:20:13.328618 23686'267906 23686:1593362 [59,168] 59 [59,168] 59 23662'267319 2022-12-18 07:20:13.328540 23662'267152 2022-12-13 19:52:53.012182 0
3.e8a 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-28 13:45:09.179473 0'0 23686:18789 [108,64] 108 [108,64] 108 0'0 2022-12-28 13:45:09.179409 0'0 2022-12-28 13:45:09.179409 0
1.e88 60153 0 0 0 0 177311741723 0 0 3075 3075 active+clean 2023-01-01 04:21:47.767674 23686'266189 23686:1668748 [137,81] 137 [137,81] 137 23686'266172 2023-01-01 04:21:47.767620 23686'266128 2022-12-30 22:45:30.912755 0
3.e85 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-28 13:45:12.962084 0'0 23686:13019 [112,45] 112 [112,45] 112 0'0 2022-12-28 13:45:12.962034 0'0 2022-12-28 13:45:12.962034 0
1.e87 59929 0 0 0 0 176533295427 0 0 3049 3049 active+clean 2022-12-31 18:16:00.647528 23686'264595 23686:1527102 [117,170] 117 [117,170] 117 23686'264560 2022-12-31 18:16:00.647476 23686'264494 2022-12-30 06:21:02.489588 0
3.e84 0 0 0 0 0 0 0 0 0 0 active+clean 2023-01-01 06:36:20.342735 0'0 23686:7209 [150,62] 150 [150,62] 150 0'0 2023-01-01 06:36:20.342691 0'0 2022-12-20 01:13:12.470701 0
1.e86 60155 0 0 0 0 177238192213 0 0 3076 3076 active+clean 2022-12-31 23:29:33.487171 23686'267316 23686:1251745 [64,42] 64 [64,42] 64 23686'267298 2022-12-31 23:29:33.487126 23686'267298 2022-12-31 23:29:33.487126 0
3.e87 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 18:38:55.362599 0'0 23686:22345 [21,171] 21 [21,171] 21 0'0 2022-12-31 18:38:55.362538 0'0 2022-12-20 02:27:20.505038 0
1.e85 59960 0 0 0 0 176462625858 0 0 3039 3039 active+clean 2022-12-28 14:12:58.614834 23686'267960 23686:1703075 [178,3] 178 [178,3] 178 23686'267812 2022-12-28 14:12:58.614794 23686'267812 2022-12-28 14:12:58.614794 0
3.e86 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 21:23:52.856914 0'0 23686:25444 [64,144] 64 [64,144] 64 0'0 2022-12-31 21:23:52.856861 0'0 2022-12-23 05:07:09.853120 0
1.e84 60382 0 0 0 0 177545253581 0 0 3052 3052 active+clean 2022-12-31 10:34:48.971952 23686'265991 23686:1470790 [118,33] 118 [118,33] 118 23686'265947 2022-12-31 10:34:48.971907 23686'265806 2022-12-28 11:10:20.014077 0
3.e81 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-28 13:45:13.079440 0'0 23686:11568 [141,72] 141 [141,72] 141 0'0 2022-12-28 13:45:13.079391 0'0 2022-12-28 13:45:13.079391 0
1.e83 59549 0 0 0 0 174492785801 0 0 3055 3055 active+clean 2023-01-01 05:30:29.513513 23686'264403 23686:1357982 [47,39] 47 [47,39] 47 23686'264396 2023-01-01 05:30:29.513468 23686'264350 2022-12-31 00:37:11.780263 0
3.e80 0 0 0 0 0 0 0 0 0 0 active+clean 2022-12-31 12:12:35.898591 0'0 23686:18485 [93,125] 93 [93,125] 93 0'0 2022-12-31 12:12:35.898517 0'0 2022-12-25 03:23:24.868798 0
1.e82 59984 0 0 0 0 176200333351 0 0 3003 3003 active+clean+scrubbing+deep 2023-01-01 11:11:24.225656 23686'266139 23686:1668141 [102,131] 102 [102,131] 102 23662'265369 2022-12-18 07:17:59.598616 23662'265369 2022-12-18 07:17:59.598616 0
Thanks for any help,
jeff
--
Jeff Turmelle
International Research Institute for Climate & Society
Columbia University Climate School
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
