Hi all,
We started running a EC pool based object store, set up with a 4+2 configuration, and we seem to be getting an almost constant report of inconsistent PGs during scrub operations. For example:
root@rook-tools:/# ceph pg ls inconsistent
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP
19.26 104 0 0 0 0 436207616 1537 1537 active+clean+inconsistent 2018-06-28 15:04:54.054227 1811'3137 2079:5075 [206,116,68,31,193,156] 206 [206,116,68,31,193,156]
206 1811'3137 2018-06-27 21:00:17.011611 1811'3137 2018-06-27 21:00:17.011611 19.234 98 0 0 0 0 406847488 1581 1581 active+clean+inconsistent 2018-06-28 15:05:18.077003 1811'2981 2080:4822 [28,131,229,180,84,68] 28 [28,131,229,180,84,68]
28 1811'2981 2018-06-28 14:09:54.092401 1811'2981 2018-06-28 14:09:54.092401 19.2a8 116 0 0 0 0 486539264 1561 1561 active+clean+inconsistent 2018-06-28 15:04:54.073762 1811'3161 2079:4825 [177,68,222,13,131,107] 177 [177,68,222,13,131,107]
177 1811'3161 2018-06-28 07:51:21.109587 1811'3161 2018-06-28 07:51:21.109587 19.406 126 0 0 0 0 520233399 1557 1557 active+clean+inconsistent 2018-06-28 15:04:57.142651 1811'3057 2080:4944 [230,199,128,68,92,11] 230 [230,199,128,68,92,11]
230 1811'3057 2018-06-27 18:36:18.497899 1811'3057 2018-06-27 18:36:18.497899 19.46b 109 0 0 0 0 449840274 1558 1558 active+clean+inconsistent 2018-06-28 15:04:54.227970 1811'3058 2079:4986 [18,68,130,94,181,225] 18 [18,68,130,94,181,225]
18 1811'3058 2018-06-27 14:32:17.800961 1811'3058 2018-06-27 14:32:17.800961 [snip]
We sometimes see that running a deep scrub on the PG resolves the issue but not all the time.
We have been running the PG repair operation on them (e.g. ceph pg repair <pg num>), which clears the issue. Is this the correct way to resolve this issue?
Is this a normal behavior for a Ceph cluster?
If so, why doesn’t it attempt to repair itself automatically?
Thanks for the help understanding Ceph, we are very new to it still!
-Bryan
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential, or privileged information and/or personal data. If you are not the intended recipient, you are hereby notified that any review, dissemination, or copying of this email is strictly prohibited, and requested to notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request, or solicitation of any kind to buy, sell, subscribe, redeem, or perform any type of transaction of a financial product. Personal data, as defined by applicable data privacy laws, contained in this email may be processed by the Company, and any of its affiliated or related companies, for potential ongoing compliance and/or business-related purposes. You may have rights regarding your personal data; for information on exercising these rights or the Company’s treatment of personal data, please email datarequests@xxxxxxxxxxxxxxx.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
--
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
