On Wed, Jun 20, 2018 at 6:37 AM, John Spray <jspray@xxxxxxxxxx> wrote: > On Tue, Jun 19, 2018 at 8:02 PM Sage Weil <sage@xxxxxxxxxx> wrote: >> >> I've seen a couple of cases in the last two weeks where users have removed >> OSDs from their clusters too soon and either lost data or had to scramble >> to re-add OSDs. I'm wondering how we can improve the CLI/tools to make >> this situation harder. >> >> In luminous, we added a few new commands: >> >> - ceph osd destroy: zap info about an OSD but keep it's ID in place (with >> a 'destroyed' flag) so that it can be recreated with a replacement device. >> - ceph osd purge: zap everything about an OSD, including the ID >> >> Once these commands are run it is hard to re-add the OSD device back into >> the cluster if the operator realizes there are PGs stuck peering or >> unfound objects. >> >> There are two other new commands: >> >> - ceph osd ok-to-stop: Checks whether it looks like PGs will remain >> available even if the specified OSD(s) are stopped. >> - ceph osd safe-to-destroy: Checks whether it is safe to destroy an OSD. >> This does various checks to ensure there is no data on the OSD(s), no >> unfound objects, stuck peering, and so forth. >> >> Once could argue that the users who got into trouble should have run >> 'ceph osd safe-to-destory' before removing the devices. That should have >> avoided their problem, but we should do what we can to make it hard >> for them to *not* see the safety check. >> >> So, two ideas: >> >> First, let's get rid of 'ceph osd rm'. Users should use destroy or purge; >> this command only does part of the job and has an unclear purpose >> now that the others exist. I'll probably sound like an outlier here, but I swear I always get the outcome of purge and destroy wrong and have to go and check what they do :( > > +1 > >> Second, change destroy and purge to do the safe-to-destroy check. If the >> check passes, do the removal. If the check fails, issue a warning (with >> some detail in the error string) and require the --yes-i-really-mean-it >> option. (Currently, these commands *always* require the force flag >> but don't perform the safety check.) > > Yes, and in the case where the check fails, we can be helpful by > perhaps marking the OSD out to prompt data to migrate away, and/or > give a message explaining "there's X MB of data on here, but we're > migrating it away (at X objects per second)". > > This is a bit OT, but I'm a bit dubious about continuing to use > "--yes-i-really-mean-it" flags instead of a simpler "--force" flag. > The yes-i-really-mean-it sometimes feels like a Ceph in-joke, when > most other software calls it --force or similar. Force is maybe a bit > too easy to type, but a lot of our users seem to have > yes-i-really-mean-it in their muscle memory too :-/ I agree with this, --force is commonly accepted in other software to prevent refusals or prompts. I also don't think that the longer the flag the less prone a user is going to make a mistake. > >> The main downside to this is that current scripts (well, post-luminous >> scripts) may include the force flag (since it used to be unconditionally >> required) and miss the safety check. On the other hand, they are already >> ignore the safety check, so they are no worse off... they just see no >> benefit. > > I agree -- any script which is trying to do a purge unconditionally > will get what they asked for! In ceph-ansible, and in some (small) portions of ceph-volume, we use --yes-i-really-mean-it, and I think we would prefer the unconditional behavior to stay. I think something of value could be to add the warning message from the failed check *even when* the force flag was used. > > John > >> >> Thoughts? >> sage >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
