One other oddity I've found is that ceph left 51 GB of data on each of the OSDs on the retired hardware. Is that by design or could it indicate some other problems? The PGs there seem to now be remapped elsewhere.....
Regards,
Jeff
On Tue, Mar 8, 2016 at 2:09 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
The pgs are not actually inconsistent (that is, I think that all of
the real objects are present and healthy). I think each of those pgs
has one of these duplicate pairs confusing scrub (and also pg removal
-- hence your ENOTEMPTY bug). Once we figure out what's going on,
you'll have to clean them up manually. Do not repair any of these. I
suggest that you simply disable scrub and ignore the inconsistent flag
until we have an idea of what is going on.
-Sam
On Tue, Mar 8, 2016 at 12:06 PM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote:
> I restarted the OSDs with the 'unfound' objects and now I have none, but I
> have 43 inconsistent PGs that I need to repair.....I only see unfound files
> once issue the 'pg repair'. How do I clear out the inconsistent states?
>
>
> ceph -s
> cluster 5221cc73-869e-4c20-950f-18824ddd6692
> health HEALTH_ERR
> 43 pgs inconsistent
> 3507 scrub errors
> noout flag(s) set
> monmap e9: 3 mons at
> {cephmon1=10.32.16.93:6789/0,cephmon2=10.32.16.85:6789/0,cephmon3=10.32.16.89:6789/0}
> election epoch 112718, quorum 0,1,2 cephmon2,cephmon3,cephmon1
> mdsmap e11408: 1/1/1 up {0=0=up:active}
> osdmap e279630: 449 osds: 449 up, 422 in
> flags noout
> pgmap v26505719: 7788 pgs, 21 pools, 251 TB data, 88784 kobjects
> 412 TB used, 2777 TB / 3190 TB avail
> 7731 active+clean
> 43 active+clean+inconsistent
> 7 active+clean+scrubbing+deep
> 7 active+clean+scrubbing
>
> Jeff
>
> On Tue, Mar 8, 2016 at 2:00 PM, Samuel Just <sjust@xxxxxxxxxx> wrote:
>>
>> Yeah, that procedure should have isolated any filesystem issues. Are
>> there still unfound objects?
>> -sam
>>
>
> --
>
> Jeffrey McDonald, PhD
> Assistant Director for HPC Operations
> Minnesota Supercomputing Institute
> University of Minnesota Twin Cities
> 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx
> 117 Pleasant St SE phone: +1 612 625-6905
> Minneapolis, MN 55455 fax: +1 612 624-8861
>
>
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
