Would you be able to execute ``ceph pg ${PG ID} query`` against that
particular PG?
On Wed, Dec 21, 2016 at 11:44 PM, Andras Pataki
<apataki@xxxxxxxxxxxxxxxxxxxx> wrote:
> Yes, size = 3, and I have checked that all three replicas are the same zero
> length object on the disk. I think some metadata info is mismatching what
> the OSD log refers to as "object info size". But I'm not sure what to do
> about it. pg repair does not fix it. In fact, the file this object
> corresponds to in CephFS is shorter so this chunk shouldn't even exist I
> think (details are in the original email). Although I may be understanding
> the situation wrong ...
>
> Andras
>
>
> On 12/21/2016 07:17 AM, Mehmet wrote:
>
> Hi Andras,
>
> Iam not the experienced User but i guess you could have a look on this
> object on each related osd for the pg, compare them and delete the Different
> object. I assume you have size = 3.
>
> Then again pg repair.
>
> But be carefull iirc the replica will be recovered from the primary pg.
>
> Hth
>
> Am 20. Dezember 2016 22:39:44 MEZ, schrieb Andras Pataki
> <apataki@xxxxxxxxxxxxxxxxxxxx>:
>>
>> Hi cephers,
>>
>> Any ideas on how to proceed on the inconsistencies below? At the moment
>> our ceph setup has 5 of these - in all cases it seems like some zero length
>> objects that match across the three replicas, but do not match the object
>> info size. I tried running pg repair on one of them, but it didn't repair
>> the problem:
>>
>> 2016-12-20 16:24:40.870307 7f3e1a4b1700 0 log_channel(cluster) log [INF]
>> : 6.92c repair starts
>> 2016-12-20 16:27:06.183186 7f3e1a4b1700 -1 log_channel(cluster) log [ERR]
>> : repair 6.92c 6:34932257:::1000187bbb5.00000009:head on disk size (0) does
>> not match object info size (3014656) adjusted for ondisk to (3014656)
>> 2016-12-20 16:27:35.885496 7f3e17cac700 -1 log_channel(cluster) log [ERR]
>> : 6.92c repair 1 errors, 0 fixed
>>
>>
>> Any help/hints would be appreciated.
>>
>> Thanks,
>>
>> Andras
>>
>>
>> On 12/15/2016 10:13 AM, Andras Pataki wrote:
>>
>> Hi everyone,
>>
>> Yesterday scrubbing turned up an inconsistency in one of our placement
>> groups. We are running ceph 10.2.3, using CephFS and RBD for some VM
>> images.
>>
>> [root@hyperv017 ~]# ceph -s
>> cluster d7b33135-0940-4e48-8aa6-1d2026597c2f
>> health HEALTH_ERR
>> 1 pgs inconsistent
>> 1 scrub errors
>> noout flag(s) set
>> monmap e15: 3 mons at
>> {hyperv029=10.4.36.179:6789/0,hyperv030=10.4.36.180:6789/0,hyperv031=10.4.36.181:6789/0}
>> election epoch 27192, quorum 0,1,2
>> hyperv029,hyperv030,hyperv031
>> fsmap e17181: 1/1/1 up {0=hyperv029=up:active}, 2 up:standby
>> osdmap e342930: 385 osds: 385 up, 385 in
>> flags noout
>> pgmap v37580512: 34816 pgs, 5 pools, 673 TB data, 198 Mobjects
>> 1583 TB used, 840 TB / 2423 TB avail
>> 34809 active+clean
>> 4 active+clean+scrubbing+deep
>> 2 active+clean+scrubbing
>> 1 active+clean+inconsistent
>> client io 87543 kB/s rd, 671 MB/s wr, 23 op/s rd, 2846 op/s wr
>>
>> # ceph pg dump | grep inconsistent
>> 6.13f1 4692 0 0 0 0 16057314767 3087 3087
>> active+clean+inconsistent 2016-12-14 16:49:48.391572 342929'41011
>> 342929:43966 [158,215,364] 158 [158,215,364] 158 342928'40540
>> 2016-12-14 16:49:48.391511 342928'40540 2016-12-14 16:49:48.391511
>>
>> I tried a couple of other deep scrubs on pg 6.13f1 but got repeated
>> errors. In the OSD logs:
>>
>> 2016-12-14 16:48:07.733291 7f3b56e3a700 -1 log_channel(cluster) log [ERR]
>> : deep-scrub 6.13f1 6:8fc91b77:::1000187bb70.00000009:head on disk size (0)
>> does not match object info size (1835008) adjusted for ondisk to (1835008)
>> I looked at the objects on the 3 OSD's on their respective hosts and they
>> are the same, zero length files:
>>
>> # cd ~ceph/osd/ceph-158/current/6.13f1_head
>> # find . -name *1000187bb70* -ls
>> 669738 0 -rw-r--r-- 1 ceph ceph 0 Dec 13 17:00
>> ./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.00000009__head_EED893F1__6
>>
>> # cd ~ceph/osd/ceph-215/current/6.13f1_head
>> # find . -name *1000187bb70* -ls
>> 539815647 0 -rw-r--r-- 1 ceph ceph 0 Dec 13 17:00
>> ./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.00000009__head_EED893F1__6
>>
>> # cd ~ceph/osd/ceph-364/current/6.13f1_head
>> # find . -name *1000187bb70* -ls
>> 1881432215 0 -rw-r--r-- 1 ceph ceph 0 Dec 13 17:00
>> ./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.00000009__head_EED893F1__6
>>
>> At the time of the write, there wasn't anything unusual going on as far as
>> I can tell (no hardware/network issues, all processes were up, etc).
>>
>> This pool is a CephFS data pool, and the corresponding file (inode hex
>> 1000187bb70, decimal 1099537300336) looks like this:
>>
>> # ls -li chr4.tags.tsv
>> 1099537300336 -rw-r--r-- 1 xichen xichen 14469915 Dec 13 17:01
>> chr4.tags.tsv
>>
>> Reading the file is also ok (no errors, right number of bytes):
>> # cat chr4.tags.tsv > /dev/null
>> # wc chr4.tags.tsv
>> 592251 2961255 14469915 chr4.tags.tsv
>>
>> We are using the standard 4MB block size for CephFS, and if I interpret
>> this right, this is the 9th chunk, so there shouldn't be any data (or even a
>> 9th chunk), since the file is only 14MB. Should I run pg repair on this?
>> Any ideas on how this could come about? Any other recommendations?
>>
>> Thanks,
>>
>> Andras
>> apataki@xxxxxxxxxxx
>>
>>
>> ________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
