lost objects

[Jeffrey McDonald <jmcdonal@xxxxxxx>]

Hi,

I have a question about lost objects.   I have a placement group in the active+recovering state with a missing object.    Should I wait until the placement group is  active+clean before doing anything about the missing objects?   

# ceph -v
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)

#  ceph health
HEALTH_ERR 67 pgs inconsistent; 48 pgs recovering; 48 pgs stuck unclean; 5 requests are blocked > 32 sec; recovery 24677/534737733 objects degraded (0.005%); recovery 1011/90652026 unfound (0.001%); 5537 scrub errors; noout flag(s) set

Whe I query a pg with a missing object, I see:

{
    "offset": {
        "oid": "",
        "key": "",
        "snapid": 0,
        "hash": 0,
        "max": 0,
        "pool": -1,
        "namespace": ""
    },
    "num_missing": 1,
    "num_unfound": 1,
    "objects": [
        {
            "oid": {
                "oid": "default.325674.107__shadow_PSI_COMMON\/DATA\/REFLECTANCE_DAILY\/DOTMATs\/data_MOD09GA_h10v05_500m_daily_2003.mat.2~skoVlnbFGILyQRSgfrvjDHIvE8Mm1gT.10_111",
                "key": "",
                "snapid": -2,
                "hash": 4261306676,
                "max": 0,
                "pool": 70,
                "namespace": ""
            },
            "need": "73544'201734",
            "have": "0'0",
            "locations": [
                "322(2)"
            ]
        }
    ],
    "more": 0
}

Is there a way to tell if the object can be reverted or if it is a new object that is now lost? 

The recovery state of the placement group is maybe not encouraging (because it shows the OSD are already probed--if I interpret this correctly):

"recovery_state": [
        {
            "name": "Started\/Primary\/Active",
            "enter_time": "2016-02-05 08:25:16.558771",
            "might_have_unfound": [
                {
                    "osd": "7(3)",
                    "status": "already probed"
                },
                {
                    "osd": "17(5)",
                    "status": "already probed"
                },
                {
                    "osd": "43(1)",
                    "status": "already probed"
                },
                {
                    "osd": "172(4)",
                    "status": "already probed"
                },
                {
                    "osd": "322(2)",
                    "status": "already probed"
                }
            ],
            "recovery_progress": {
                "backfill_targets": [],
                "waiting_on_backfill": [],
                "last_backfill_started": "0\/\/0\/\/-1",
                "backfill_info": {
                    "begin": "0\/\/0\/\/-1",
                    "end": "0\/\/0\/\/-1",
                    "objects": []
                },
                "peer_backfill_info": [],
                "backfills_in_flight": [],
                "recovering": [],
                "pg_backend": {
                    "recovery_ops": [],
                    "read_ops": []
                }
            },
            "scrub": {
                "scrubber.epoch_start": "143439",
                "scrubber.active": 0,
                "scrubber.waiting_on": 0,
                "scrubber.waiting_on_whom": []
            }
        },
        {
            "name": "Started",
            "enter_time": "2016-02-05 08:22:37.288281"
        }

Thanks,

Jeff

--

Jeffrey McDonald, PhD
Assistant Director for HPC Operations
Minnesota Supercomputing Institute
University of Minnesota Twin Cities
599 Walter Library           email: jeffrey.mcdonald@xxxxxxxxxxx
117 Pleasant St SE           phone: +1 612 625-6905
Minneapolis, MN 55455        fax:   +1 612 624-8861

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com