Re: inconsistent pg issue with ceph version 10.2.3

[Thanh Tran <cephvn@xxxxxxxxx>]

I fixed this inconsistent error. It seems ceph didn't delete the mismatch object that depends the deleted snapshot. This caused error "unexpected clone" that resulted of message inconsistent.

Log:

2017-12-12 20:14:06.651942 7fc7eff7e700 -1 log_channel(cluster) log [ERR] : deep-scrub 4.1b42 4:42db3cde:::rbd_data.aa5af9238e1f29.000000000000470a:2ccac is an unexpected clone

Steps to fix:

1. Move the mismatch object on both of osd.159 and osd.179 to other location

2. ceph pg deep-scrub 4.1b42

On Tue, Dec 12, 2017 at 9:34 PM, Thanh Tran <cephvn@xxxxxxxxx> wrote:

Hi,

My ceph cluster has a inconsistent pg. I tried to deep scrub pg and repair pg but not fix problems.

I found that the object that made pg inconsistent depends on snapshot (snap id is 2ccac = 183468) of a image, I deleted this snapshot, then query inconsistent pg and it showed empty, but my ceph cluster still show error inconsistent pg. It's strange that the mismatch object depends on the deleted snapshot still exist on disk.

After deleting the snapshot has the mismatch object, I re-tried to deep scrub and repair pg but it doesn't help to fix.

Information of commands as below. I don't know how to clear this status of inconsistent. Please help me to fix this bug.

ceph --cluster QTC01 health detail

HEALTH_ERR 1 pgs inconsistent; 1 scrub errors

pg 4.1b42 is active+clean+inconsistent, acting [159,179]

1 scrub errors

rados list-inconsistent-obj 4.1b42 --format=json-pretty --cluster QTC01

{

    "epoch": 494830,

    "inconsistents": []

}

ll /var/lib/ceph/osd/QTC01-159/current/4.1b42_head/DIR_2/DIR_4/DIR_B/ | grep aa5af9238e1f29

-rw-r--r-- 1 ceph ceph 4194304 Dec 12 15:29 rbd\udata.aa5af9238e1f29.000000000000470a__2ccac_7B3CDB42__4

-rw-r--r-- 1 ceph ceph 1048576 Dec 12 15:29 rbd\udata.aa5af9238e1f29.000000000000470a__head_7B3CDB42__4

ll /var/lib/ceph/osd/QTC01-179/current/4.1b42_head/DIR_2/DIR_4/DIR_B/ | grep aa5af9238e1f29

-rw-r--r-- 1 ceph ceph 4194304 Dec 12 15:17 rbd\udata.aa5af9238e1f29.000000000000470a__2ccac_7B3CDB42__4

-rw-r--r-- 1 ceph ceph 1048576 Dec 12 15:17 rbd\udata.aa5af9238e1f29.000000000000470a__head_7B3CDB42__4

Best regards,

Thanh Tran

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com