Re: backfill_unfound state reset to clean after osd restart

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, May 12, 2021 at 10:10:45AM +0300, Mykola Golub wrote:

> > So, the current behavior is: In `PrimaryLogPG::recover_backfill`, due
> > to the "unfound" oid is not removed from `backfills_in_flight`,
> > `next_backfill_to_complete` is always set to the "unfound" oid [4],
> > and `new_last_backfill` is not updated any more pointing to the object
> > before the "unfound" oid. The backfill still continues and terminates
> > only after all objects are pulled/pushed, but "complete" position
> > remains on the object before "unfound". After the backfill is finished
> > the pg enters "backfill_unfound" state. When the pg is re-peered
> > (e.g. after restarting an osd) it enters "backfilling" state starting
> > the backfill from "unfound" oid position, detects the "unfound" object
> > again, scans the remaining objects detecting they are already copied,
> > and enters "backfill_unfound" state again with the same "complete"
> > position on the "unfound" object.
> > 
> > This looks like a reasonable behavoir to me, and the only problem is
> > that reported assertion failure, which probably is just needed to be
> > removed?
> 
> https://github.com/ceph/ceph/pull/41270

This PR is merged. Thanks! And "backfill_unfound" state is not reset
after a non-primary OSD restart. But it is still possible to reset the
state to clean when the primary osd is restarted.

I will re-describe the situation.

Suppose we have a 2+1 EC pool, and an object is missing 2 shards on
both non-primary osds. We initiate backfill by setting a non-primary
osd out. During the backfill the primary osd detects the missing
shards and the pg enters "backfill_unfound" state, the last_backfill
position is properly set to the object before the "unfound" (in
post-nautilus, for nautilus I opened [1] to make it work). If
re-peering occurs due to a non-primary osd is restarted, the backfill
is restarted from the last_backfill position and the "unfound" object
is detected again. But if re-peering occurs due the primary osd is
temporarily stopped (restarted), another non-primary osd becomes
primary and "drives" the backfill from the last_backfill position, and
as the shard is missing here it is just skipped from the backfill, the
missing object is not detected and the pg enters clean state.

Is there something that can/should be improved here? It is rather
unfortunate that the information about missing object is lost on the
restart (until scrub or next backfill). On the other hand the
situation when we have many shards are missing for an object is rather
unlikely. Also, if for example it happened that the shard was missing
on the primary it would not even be detected on backfill.

[1] https://github.com/ceph/ceph/pull/41293

-- 
Mykola Golub
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx



[Index of Archives]     [CEPH Users]     [Ceph Devel]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux