On Thu, Mar 29, 2018 at 6:24 AM, kefu chai <tchaikov@xxxxxxxxx> wrote: > On Tue, Mar 27, 2018 at 5:43 PM, John Spray <jspray@xxxxxxxxxx> wrote: >> On Tue, Mar 27, 2018 at 10:26 AM, kefu chai <tchaikov@xxxxxxxxx> wrote: >>> On Tue, Mar 20, 2018 at 6:45 AM, Sage Weil <sweil@xxxxxxxxxx> wrote: >>>> On Mon, 19 Mar 2018, Gregory Farnum wrote: >>>>> On Mon, Mar 19, 2018 at 7:33 AM, John Spray <jspray@xxxxxxxxxx> wrote: >>>>> > Hi all, >>>>> > >>>>> > I was looking at places in ceph-mgr where we send a command from a >>>>> > module, and then want to proceed with some logic that involves reading >>>>> > the osdmap (there is a local copy in the manager, maintained by >>>>> > Objecter). >>>>> > >>>>> > I had been thinking that we should include cluster map epochs in the >>>>> > MMonCommandAck messages so that the client can (optionally) wait for >>>>> > that latest OSDMap before it considers the command complete. >>>>> > >>>>> > Then I thought, maybe this isn't necessary at all, because the mons >>>>> > would be doing the check_subs() etc calls before they actually respond >>>>> > to commands, so clients would always get their updated maps before >>>>> > seeing a command response message. >>>>> > >>>>> > So: mon experts, what do you think? Is it safe to assume that clients >>>>> > will get their subscription updates before a command completion (even >>>>> > in the case of commands being forwarded)? Or do we maybe need a >>>>> > little bit more logic on the client side in the manager? >>>>> >>>>> I would expect the order of op replies versus subscription fulfillment >>>>> messages to be an implementation detail, even if we do currently spool >>>>> off new map subscription requests inline with committing them. (I >>>>> don’t know at all if that’s the case.) >>>> >>>> Currently all of the subs are satisfied by update_from_paxos(), which >>>> means they get fulfilled before any replies (which are waiting_for_commit >>>> completions). Having recently fixed one of the monitor services to do this >>>> that wasn't in order to fix a subsrciption bug, I'm pretty confident this >>>> is the "right" place to do it given how the mon is currently structured. >>> >>> if the command *updates* the status of monitor in the sense that it triggers a >>> proposal, i think it's safe to assume that the client which sends the command >>> will be updated with the latest osdmap. but if it just *queries* the >>> cluster status >>> from the mon, and the behavior of client depends on the osdmap, there is >>> a risk of racing. John, what specific ceph-mgr module or calling path was you >>> looking at? >> >> I was thinking specifically of updates. This was in some code I'm >> working on that creates pools: need to make sure that after my "osd >> pool create" command is done, it'll be reflected in the local OSDMap. >> >> It seems like the consensus is that it would be worthwhile to include >> map epochs in the command response for commands that update things. >> > > yeah, i feel the same. Since Sage was inclined the other way, it’s worth calling out that the opportunity cost of handling map epochs seems to be negligible. And if we don’t do that work, any future change to the monitors will require special handling for old clients. :) -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
