On Thu, 16 Feb 2017, Wido den Hollander wrote: > > Op 16 februari 2017 om 17:20 schreef Sage Weil <sage@xxxxxxxxxxxx>: > > > > On Thu, 16 Feb 2017, Sage Weil wrote: > > > There is another step here (see my other email) where we should mark the > > > osd as lost before we allow it to be replaced. So, > > > > > > 0. 'ceph osd lost NNN' from a client.admin node > > > > > > Assuming that is done, then I think the rest of the procedure would be > > > > > > 2. Try to unmount the disk (fails if OSD is still running) > > > 3. zap the disk > > > 4. ceph-disk prepare --replace-osd-id NNN > > > > > > This would do 'ceph osd replace <osd-id> <new-uuid>' instead of 'ceph > > > osd create <uuid>'. The new mon command would verify that (1) the osd is > > > marked as lost (safety check that makes bootstraps ability to do this > > > reasonably secure) and (2) change the uuid to new-uuid. It could also (3) > > > remove old cephx keys. Note that we had a thread about making create do > > > this a month or two ago; this might be a good time to fix that too. The > > > idea was that the boostrap permissions are super wonky because they have > > > to allow creating new cephx keys and so on. Instead, we should make a > > > single command that does everything (including creating the cephx keys) > > > and returns the whole result to ceph-disk in a blob of json (new osd id + > > > cephx key). The replace command could work the same way (including the > > > step of removing the old key), and then the allowed commands for > > > the bootstrap key would be just 'osd create' and 'osd replace', period. > > > > Okay, I found the other thread: > > > > http://marc.info/?t=147913846400007&r=1&w=2 > > > > which is about the additional work of setting up the lockbox keys for > > dm-crypt as part of osd creation/bootstrap. I think the way to do this > > properly is to just bite the bullet and make a new command, let's call it > > 'osd bootstrap', and have it take all the various [optional] arguments for > > setting up a new osd, including the osd we want to replace (if any). It > > can do all the right things as far as setting up (or replacing) cephx > > keys, and be a single atomic and idempotent operation so that ceph-disk is > > super simple and the osd-bootstrap key can actually be secure. > > > > This has totally blown up in scope...are you up for it? In the meantime, > > the zap fix is small and simple and unrelated to the rest. > > > > Well, yes, that's a bit more work :) I've never worked on that part of > the code in the MONs, so that will be new for me. Could be a good thing > to try and learn from. I wrote up some notes on an etherpad: http://pad.ceph.com/p/osd-replacement A few changes: - I think 'ceph osd rm' would make more sense than 'ceph osd lost' for the safety gate. - The 'osd create' replacement command has some notes. It includes the ability to set some arbitrary config-key values so that it can be used for the dmcrypt lockbox (or hopefully future things too). sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
