On Tue, Aug 26, 2014 at 10:46 PM, John Morris <john at zultron.com> wrote: > In the docs [1], 'incomplete' is defined thusly: > > Ceph detects that a placement group is missing a necessary period of > history from its log. If you see this state, report a bug, and try > to start any failed OSDs that may contain the needed information. > > However, during an extensive review of list postings related to > incomplete PGs, an alternate and oft-repeated definition is something > like 'the number of existing replicas is less than the min_size of the > pool'. In no list posting was there any acknowledgement of the > definition from the docs. > > While trying to understand what 'incomplete' PGs are, I simply set > min_size = 1 on this cluster with incomplete PGs, and they continue to > be 'incomplete'. Does this mean that definition #2 is incorrect? > > In case #1 is correct, how can the cluster be told to forget the lapse > in history? In our case, there was nothing writing to the cluster > during the OSD reorganization that could have caused this lapse. Yeah, these two meanings can (unfortunately) both lead to the INCOMPLETE state being reported. I think that's going to be fixed in our next major release (so that INCOMPLETE means "not enough OSDs hosting" and "missing log" will translate into something else), but for now the "not enough OSDs" is by far the more common. In your case you probably are missing history, but you don't want to recover from it using any of the cluster tools because they're likely to lose more data than necessary. (Hopefully, you can just roll back to a slightly older btrfs snapshot, but we'll see). -Greg
 |