On Wed, 1 Nov 2017, Gregory Farnum wrote: > On Wed, Oct 25, 2017 at 7:43 AM Joao Eduardo Luis <joao@xxxxxxx> wrote: > > > > On 10/24/2017 03:36 PM, Ricardo Dias wrote: > > > Hi list, > > > > > > I was wondering if it makes sense to support multiple binds with a > > > single messenger instance. > > > > > > The use case I have in mind is when there are multiple public/cluster > > > networks specified in ceph.conf and we want to listen for connections > > > in one interface of each network. With the support for multiple binds, > > > we could use a single messenger instance to listen to all interfaces. > > > > > > Do you think it is worth to implement such support or is the above use > > > case easily handled by having multiple messenger instances? > > > > I would think multiple binds would make sense, but how would this affect > > setting the policies? Would we be sharing a single policy across > > multiple binds, or would we be allowed different policies for different > > binds? > > > > Regardless, and not knowing the code, wouldn't this make sense for > > multiplexing multiple daemons on a single messenger? Making the > > messenger sort of a blackbox, that everyone could just use seamlessly > > without having to figure too many things out? > > > > E.g., instead of multiplexing several osds on a messenger for the public > > network, and then the same set of osds on a messenger for the cluster > > network, we could simply have one messenger (bound to public and > > cluster) and have just that one messenger for all the osds. > > > > Anyway, from an abstraction point-of-view it makes sense to me; no idea > > how feasible that would be though. > > There's sort of two pieces here, right? > 1) From the Dispatcher perspective, we want a single "Messenger" that > it talks to. (At least, I'm pretty sure. Unless we need to be making > routing decisions using internal data? I really hope that never comes > up.) That could be set up by having multiple Messenger implementations > behind a "UnifyingMessenger" abstraction layer with pretty minimal > changes. > > 2) From an implementation perspective, which data structures do we end > up needing to share in order to have a daemon listening on multiple > ports? > > So, from the SimpleMessenger perspective, I don't think we get much > out of trying to build multi-bind into it. We'd basically get to > eliminate the duplicated DIspatchQueue, which is not very interesting. > (Modulo some flow control stuff, perhaps.) From an AsyncMessenger > implementation perspective, that calculus might shift since we're > running on a thread pool, but I'm not deeply familiar with the code > internals. I'm afraid that the problem that's going to come up is the peer_addr hash_map<> for stateful_peer connections (osd to osd, etc.). This is down in the messenger layer and is the main piece of shared state left (most everything else works with Connection refs). I'm worried that we'll want or need to pull that up out of the messenger into the consumer (e.g., a hash_map<int,ConnectionRef> osd_peers in OSDService) and refactor the connection race handling a bit. I think that will ultimately solve some bugs and will simplify the Messenger greatly, but it's a painful refactor. > But I'm curious what brings this up. I thought you were working on the > v2 protocol — does something in that somehow hinge on the server-side > implementation of multiple networks? The v2 protocol will run on another port and we want to answer on the old port+protocol for old clients too. sage
