Just for clarification, the PG state is not the cause of the scrub errors. Something happened in your cluster that caused inconsistencies between copies of the data, the scrub noticed them, the scrub errors are why the PG is flagged inconsistent, which does put the cluster in HEALTH_ERR. Anyway, just semantics from your original assessment of the situation.
Disabling scrubs is a bad idea here. While you have a lot of scrub errors, you only know of 1 PG that has those errors. You may have multiple PGs with the same problem. Perhaps a single disk is having problems and every PG on that disk has scrub errors. There are a lot of other scenarios that could be happening as well. I would start by issuing `ceph osd scrub $osd` to scrub all PGs on the currently known OSDs used by this PG. If that doesn't find anything, then try `ceph osd deep-scrub $osd`. Those commands are a shortcut to schedule a scrub/deep-scrub for every PG that is primary on the given OSD. If you don't find any more scrub errors, then you may need to check the rest of the PGs in your cluster, definitely the ones inside of the same pool #2 along with the currently inconsistent PG.
Now, while that's diagnosing and getting us more information... what happened to your cluster? Anything where OSDs were flapping up and down? You added new storage? Lost a drive? Upgraded versions? What is your version? What has happened in the past few weeks in your cluster?
Likely, the fix is going to start with issuing a repair of your PG. I like to diagnose the full scope of the problem before trying to repair things. Also, if I can't figure out what's going on, I try to backup the PG copies I'm repairing before doing so just in case something doesn't repair properly.
On Sat, May 12, 2018 at 2:38 AM Faizal Latif <ahmadfaizall@xxxxxxxxx> wrote:
_______________________________________________i can see most of the objects show oi_attr_missing. is there anyway that i can solved this? i believe this is the reason why scrubbing keep failing to this pg group.i have disable scrubbing since i can see there are scrub errors. i have also try to use rados command to see object status. and below are the results.Hi Guys,i need some help. i can see currently my ceph storage showing "active+clean+inconsistent". which result HEALTH_ERR state and cause scrubbing error. you may find below are sample output.
HEALTH_ERR 1 pgs inconsistent; 11685 scrub errors; noscrub,nodeep-scrub flag(s) set
pg 2.2c0 is active+clean+inconsistent, acting [28,17,37]
11685 scrub errors
noscrub,nodeep-scrub flag(s) set
rados list-inconsistent-obj 2.2c0 --format=json-pretty
{
"epoch": 57580,
"inconsistents": [
{
"object": {
"name": "rbd_data.10815ea2ae8944a.0000000000000385",
"nspace": "",
"locator": "",
"snap": 55,
"version": 0
},
"errors": [],
"union_shard_errors": [
"missing",
"oi_attr_missing"
],
"shards": [
{
"osd": 10,
"errors": [
"oi_attr_missing"
],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x32133b39"
},
{
"osd": 28,
"errors": [
"missing"
]
},
{
"osd": 37,
"errors": [
"missing"
]
}
]
},
{
"object": {
"name": "rbd_data.10815ea2ae8944a.0000000000000730",
"nspace": "",
"locator": "",
"snap": 55,
"version": 0
},
"errors": [],
"union_shard_errors": [
"missing",
"oi_attr_missing"
],
"shards": [
{
"osd": 10,
"errors": [
"oi_attr_missing"
],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x0f843f64"
},
{
"osd": 28,
"errors": [
"missing"
]
},
{
"osd": 37,
"errors": [
"missing"
]
}
]
},
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
