Hmm, so you've got PGs which are out-of-date on disk (by virtue of being an
older snapshot?) but still have records of them being newer in the OSD
journal?
That's a new failure node for me and I don't think we have any tools
designed for solving. If you can *back up the disk* before doing this, I
think I'd try flushing the OSD journal, then zapping it and creating a new
one.
You'll probably still have an inconsistent store, so you'll then need to,
for each incomplete pg (and for every OSD at the same point on it) use the
OSD store tool extract the "pginfo" entry and rewrite it to the versions
which the pg actually is on disk.
These are uncharted waters and you're going to need to explore some dev
tools; good luck!
(Unless Sam or somebody has a better solution.)
-Greg
On Thursday, August 28, 2014, John Morris <john at zultron.com> wrote:
> Greg, thanks for the tips in both this and the BTRFS_IOC_SNAP_CREATE
> thread. They were enough to get PGs 'incomplete' due to 'not enough
> OSDs hosting' resolved by rolling back to a btrfs snapshot. I promise
> to write a full post-mortem (embarrassing as it will be) after the
> cluster is fully healthy.
>
> As is my kind of luck, the cluster *also* suffers from eight of the
> *other* 'missing log' sort of 'incomplete' PGs:
>
> 2014-08-28 23:42:03.350612 7f1cc9d82700 20 osd.1 pg_epoch: 13085
> pg[2.5c( v 10143'300715 (9173'297714,10143'300715] local-les=11890
> n=3404 ec=1 les/c 11890/10167 12809/12809/12809) [7,3,0,4]
> r=-1 lpr=12809 pi=8453-12808/138 lcod 0'0
> inactive NOTIFY]
> handle_activate_map: Not dirtying info:
> last_persisted is 13010 while current is 13085
>
> The data clearly exists in the renegade OSD's data directory. There
> are no reported 'unfound' objects, so 'mark_unfound_lost revert' doesn't
> apply. No apparently useful data from 'ceph pg ... query', but an
> example is pasted below.
>
> Since the beginning of the cluster rebuild, all ceph clients have been
> turned off, so I believe there's no fear of lost data by reverting to
> these PGs, and besides there're always backup tapes.
>
> How can Ceph be told to accept the versions it sees on osd.1 as the
> most current version, and forget the missing log history?
>
> John
>
>
> $ ceph pg 2.5c query
> { "state": "incomplete",
> "epoch": 13241,
> "up": [
> 7,
> 1,
> 0,
> 4],
> "acting": [
> 7,
> 1,
> 0,
> 4],
> "info": { "pgid": "2.5c",
> "last_update": "10143'300715",
> "last_complete": "10143'300715",
> "log_tail": "9173'297715",
> "last_backfill": "0\/\/0\/\/-1",
> "purged_snaps": "[]",
> "history": { "epoch_created": 1,
> "last_epoch_started": 11890,
> "last_epoch_clean": 10167,
> "last_epoch_split": 0,
> "same_up_since": 13229,
> "same_interval_since": 13229,
> "same_primary_since": 13118,
> "last_scrub": "10029'298459",
> "last_scrub_stamp": "2014-08-18 17:36:01.079649",
> "last_deep_scrub": "8323'284793",
> "last_deep_scrub_stamp": "2014-08-15 17:38:06.229106",
> "last_clean_scrub_stamp": "2014-08-18 17:36:01.079649"},
> "stats": { "version": "10143'300715",
> "reported_seq": "1764",
> "reported_epoch": "13241",
> "state": "incomplete",
> "last_fresh": "2014-08-29 01:35:44.196909",
> "last_change": "2014-08-29 01:22:50.298880",
> "last_active": "0.000000",
> "last_clean": "0.000000",
> "last_became_active": "0.000000",
> "last_unstale": "2014-08-29 01:35:44.196909",
> "mapping_epoch": 13223,
> "log_start": "9173'297715",
> "ondisk_log_start": "9173'297715",
> "created": 1,
> "last_epoch_clean": 10167,
> "parent": "0.0",
> "parent_split_bits": 0,
> "last_scrub": "10029'298459",
> "last_scrub_stamp": "2014-08-18 17:36:01.079649",
> "last_deep_scrub": "8323'284793",
> "last_deep_scrub_stamp": "2014-08-15 17:38:06.229106",
> "last_clean_scrub_stamp": "2014-08-18 17:36:01.079649",
> "log_size": 3000,
> "ondisk_log_size": 3000,
> "stats_invalid": "0",
> "stat_sum": { "num_bytes": 0,
> "num_objects": 0,
> "num_object_clones": 0,
> "num_object_copies": 0,
> "num_objects_missing_on_primary": 0,
> "num_objects_degraded": 0,
> "num_objects_unfound": 0,
> "num_read": 0,
> "num_read_kb": 0,
> "num_write": 0,
> "num_write_kb": 0,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 0,
> "num_bytes_recovered": 0,
> "num_keys_recovered": 0},
> "stat_cat_sum": {},
> "up": [
> 7,
> 1,
> 0,
> 4],
> "acting": [
> 7,
> 1,
> 0,
> 4]},
> "empty": 0,
> "dne": 0,
> "incomplete": 1,
> "last_epoch_started": 10278},
> "recovery_state": [
> { "name": "Started\/Primary\/Peering",
> "enter_time": "2014-08-29 01:22:50.132826",
> "past_intervals": [
> [...]
> { "first": 12809,
> "last": 13101,
> "maybe_went_rw": 1,
> "up": [
> 7,
> 3,
> 0,
> 4],
> "acting": [
> 7,
> 3,
> 0,
> 4]},
> [...]
> ],
> "probing_osds": [
> 0,
> 1,
> 2,
> 3,
> 4,
> 5,
> 7],
> "down_osds_we_would_probe": [],
> "peering_blocked_by": []},
> { "name": "Started",
> "enter_time": "2014-08-29 01:22:50.132784"}]}
>
>
>
>
> On Wed, Aug 27, 2014 at 12:40 PM, Gregory Farnum <greg at inktank.com
> <javascript:;>>
> wrote:
>
> > On Tue, Aug 26, 2014 at 10:46 PM, John Morris <john at zultron.com>
> > wrote:
> > > In the docs [1], 'incomplete' is defined thusly:
> > >
> > > Ceph detects that a placement group is missing a necessary period
> > > of
> > > history from its log. If you see this state, report a bug, and try
> > > to start any failed OSDs that may contain the needed information.
> > >
> > > However, during an extensive review of list postings related to
> > > incomplete PGs, an alternate and oft-repeated definition is
> > > something
> > > like 'the number of existing replicas is less than the min_size of
> > > the
> > > pool'. In no list posting was there any acknowledgement of the
> > > definition from the docs.
> > >
> > > While trying to understand what 'incomplete' PGs are, I simply set
> > > min_size = 1 on this cluster with incomplete PGs, and they continue
> > > to
> > > be 'incomplete'. Does this mean that definition #2 is incorrect?
> > >
> > > In case #1 is correct, how can the cluster be told to forget the
> > > lapse
> > > in history? In our case, there was nothing writing to the cluster
> > > during the OSD reorganization that could have caused this lapse.
> >
> > Yeah, these two meanings can (unfortunately) both lead to the
> > INCOMPLETE state being reported. I think that's going to be fixed in
> > our next major release (so that INCOMPLETE means "not enough OSDs
> > hosting" and "missing log" will translate into something else), but
> > for now the "not enough OSDs" is by far the more common. In your case
> > you probably are missing history, but you don't want to recover from
> > it using any of the cluster tools because they're likely to lose more
> > data than necessary. (Hopefully, you can just roll back to a slightly
> > older btrfs snapshot, but we'll see).
> > -Greg
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com <javascript:;>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Software Engineer #42 @ http://inktank.com | http://ceph.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140829/b8e54243/attachment.htm>
