Chunmei, to continue the discussion in the last crimson standup, i am noting down some of our findings when reading Radoslaw's https://github.com/ceph/ceph/pull/24962. there are multiple places that we need to put the request on hold until the unmet precondition is satisfied. as noted by Yingxin that it is always more efficient to enqueue a request in application's own queue than capturing it in a continuation and stash it in the task queue of reactor. the downside of the pending request queue is that - apparently we need to maintain a queue for each precondition that could hold the requests, see the `waiting_for_*` lists/maps in PG.h. but we also need to keep track of pending futures if we want to chain the maybe_wait_*() as https://pad.ceph.com/p/crimson-io-path puts. and the pending futures are likely to be structured in very the same way of these `waiting_for_` lists, the only difference is that the value of these containers would be futures. - the op will need to go through the same checks once its precondition is satisfied and it is enqueued again. probably we need to check if there are any preconditions that implying other preconditions. if yes, is it plausible/worthy to /continue/ performing this request instead of redo all the checks ? can we reorder some of the checks for better performance? or for better readability? if we need to redo all the checks like we are doing in existing ceph-osd, we can either - use a grand seastar::repeat() to redo a request until we run into some exception or the request is served. or - use a queue for tracking the pending requests, and rerun them in the fiber that fulfill the precondition. for instance, if a batch of requests are waiting for an updated osdmap, after consuming the updated osdmap, the PG will need to serve all of the requests that are waiting for it. what do you think? -- Regards Kefu Chai
