We have three replicas, so we just performed md5sum on all of them in order to find the correct ones, then we deleted the bad file and ran pg repair.
On 15 Feb 2016 10:42 a.m., "Zoltan Arnold Nagy" <zoltan@xxxxxxxxxxxxxxxxxx> wrote:
Hi Bryan,
You were right: we’ve modified our PG weights a little (from 1 to around 0.85 on some OSDs) and once I’ve changed them back to 1, the remapped PGs and misplaced objects were gone.
So thank you for the tip.
For the inconsistent ones and scrub errors, I’m a little wary to use pg repair as that - if I understand correctly - only copies the primary PG’s data to the other PGs thus can easily corrupt the whole object if the primary is corrupted.
I haven’t seen an update on this since last May where this was brought up as a concern from several people and there were mentions of adding checksumming to the metadata and doing a checksum-comparison on repair.
Can anybody update on the current status on how exactly pg repair works in Hammer or will work in Jewel?
> On 11 Feb 2016, at 22:17, Stillwell, Bryan <bryan.stillwell@xxxxxxxxxxx> wrote:
>
> What does 'ceph osd tree' look like for this cluster? Also have you done
> anything special to your CRUSH rules?
>
> I've usually found this to be caused by modifying OSD weights a little too
> much.
>
> As for the inconsistent PG, you should be able to run 'ceph pg repair' on
> it:
>
> http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#
> pgs-inconsistent
>
>
> Bryan
>
> On 2/11/16, 11:21 AM, "ceph-users on behalf of Zoltan Arnold Nagy"
> <ceph-users-bounces@xxxxxxxxxxxxxx on behalf of zoltan@xxxxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi,
>>
>> Are there any tips and tricks around getting rid of misplaced objects? I
>> did check the archive but didn¹t find anything.
>>
>> Right now my cluster looks like this:
>>
>> pgmap v43288593: 16384 pgs, 4 pools, 45439 GB data, 10383 kobjects
>> 109 TB used, 349 TB / 458 TB avail
>> 330/25160461 objects degraded (0.001%)
>> 31280/25160461 objects misplaced (0.124%)
>> 16343 active+clean
>> 40 active+remapped
>> 1 active+clean+inconsistent
>>
>> This is how it has been for a while and I thought for sure that the
>> misplaced would converge down to 0, but nevertheless, it didn¹t.
>>
>> Any pointers on how I could get it back to all active+clean?
>>
>> Cheers,
>> Zoltan
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ________________________________
>
> This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
