I try to found data, but it's look exist with same time stamp on all osd or missing on all osd ...
So, need advice , what I need to do...
вторник, 18 августа 2015 г. пользователь Abhishek L написал:
Voloshanenko Igor writes:
> Hi Irek, Please read careful )))
> You proposal was the first, i try to do... That's why i asked about
> help... (
>
> 2015-08-18 8:34 GMT+03:00 Irek Fasikhov <malmyzh@xxxxxxxxx>:
>
>> Hi, Igor.
>>
>> You need to repair the PG.
>>
>> for i in `ceph pg dump| grep inconsistent | grep -v 'inconsistent+repair'
>> | awk {'print$1'}`;do ceph pg repair $i;done
>>
>> С уважением, Фасихов Ирек Нургаязович
>> Моб.: +79229045757
>>
>> 2015-08-18 8:27 GMT+03:00 Voloshanenko Igor <igor.voloshanenko@xxxxxxxxx>:
>>
>>> Hi all, at our production cluster, due high rebalancing ((( we have 2 pgs
>>> in inconsistent state...
>>>
>>> root@temp:~# ceph health detail | grep inc
>>> HEALTH_ERR 2 pgs inconsistent; 18 scrub errors
>>> pg 2.490 is active+clean+inconsistent, acting [56,15,29]
>>> pg 2.c4 is active+clean+inconsistent, acting [56,10,42]
>>>
>>> From OSD logs, after recovery attempt:
>>>
>>> root@test:~# ceph pg dump | grep -i incons | cut -f 1 | while read i; do
>>> ceph pg repair ${i} ; done
>>> dumped all in format plain
>>> instructing pg 2.490 on osd.56 to repair
>>> instructing pg 2.c4 on osd.56 to repair
>>>
>>> /var/log/ceph/ceph-osd.56.log:51:2015-08-18 07:26:37.035910 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> f5759490/rbd_data.1631755377d7e.00000000000004da/head//2 expected clone
>>> 90c59490/rbd_data.eb486436f2beb.0000000000007a65/141//2
>>> /var/log/ceph/ceph-osd.56.log:52:2015-08-18 07:26:37.035960 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> fee49490/rbd_data.12483d3ba0794b.000000000000522f/head//2 expected clone
>>> f5759490/rbd_data.1631755377d7e.00000000000004da/141//2
>>> /var/log/ceph/ceph-osd.56.log:53:2015-08-18 07:26:37.036133 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/head//2 expected clone
>>> fee49490/rbd_data.12483d3ba0794b.000000000000522f/141//2
>>> /var/log/ceph/ceph-osd.56.log:54:2015-08-18 07:26:37.036243 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> bac19490/rbd_data.1238e82ae8944a.000000000000032e/head//2 expected clone
>>> a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/141//2
>>> /var/log/ceph/ceph-osd.56.log:55:2015-08-18 07:26:37.036289 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> 98519490/rbd_data.123e9c2ae8944a.0000000000000807/head//2 expected clone
>>> bac19490/rbd_data.1238e82ae8944a.000000000000032e/141//2
>>> /var/log/ceph/ceph-osd.56.log:56:2015-08-18 07:26:37.036314 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/head//2 expected clone
>>> 98519490/rbd_data.123e9c2ae8944a.0000000000000807/141//2
>>> /var/log/ceph/ceph-osd.56.log:57:2015-08-18 07:26:37.036363 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> 28809490/rbd_data.edea7460fe42b.00000000000001d9/head//2 expected clone
>>> c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/141//2
>>> /var/log/ceph/ceph-osd.56.log:58:2015-08-18 07:26:37.036432 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : deep-scrub 2.490
>>> e1509490/rbd_data.1423897545e146.00000000000009a6/head//2 expected clone
>>> 28809490/rbd_data.edea7460fe42b.00000000000001d9/141//2
>>> /var/log/ceph/ceph-osd.56.log:59:2015-08-18 07:26:38.548765 7f94663b3700
>>> -1 log_channel(cluster) log [ERR] : 2.490 deep-scrub 17 errors
>>>
>>> So, how i can solve "expected clone" situation by hand?
>>> Thank in advance!
I've had an inconsistent pg once, but it was a different sort of an
error (some sort of digest mismatch, where the secondary object copies
had later timestamps). This was fixed by moving the object away and
restarting, the osd which got fixed when the osd peered, similar to what
was mentioned in Sebastian Han's blog[1].
I'm guessing the same method will solve this error as well, but not
completely sure, maybe someone else who has seen this particular error
could guide you better.
[1]: http://www.sebastien-han.fr/blog/2015/04/27/ceph-manually-repair-object/
--
Abhishek
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
