On Sun, Jan 6, 2019 at 6:30 PM Radoslaw Zarzynski <rzarzyns@xxxxxxxxxx> wrote: > > On Sat, Jan 5, 2019 at 3:42 PM kefu chai <tchaikov@xxxxxxxxx> wrote: > > > recently, when we are working on cross-core messenger[0], we found > > that, in order to share the connection between cores we need to have > > types like "seastar::lw_shared_ptr<seastar::foreign_ptr<ConnectionRef>>", > > because > > - the connections to peer OSDs are shared across cores, > > - the connections are shared by multiple continuations on the local > > core -- either locally or remotely. > > Well, I perceive this type as a clever solution to get back some of > the properties seastar::foreign_ptr had taken from us. What worries > me is that we need such heavy artillery at the very early stage of > project life. I think this is actually the key point to address. For all the potential futures we have, there are two inescapable facts: 1) a single-core OSD will be simpler to program 2) a single-core OSD will be dramatically less flexible in the face of change than anything else. So the real question boils down to: is the increased development speed worth the HUGE lock-in we get? I'm not doing much work on it yet, but I tend to think it's not worth it: 1) It's not surprising we are setting up "heavy artillery" now — this is exactly when we should be setting it up, as we establish the infrastructure needs we have which differ from what ScyllaDB needs. The messenger-PG crossbar has been much-discussed and was anticipated. 2) As I understand it, the complexity we're setting up here is really one-time programmer costs which are well-abstracted. So yes, the messenger is a little more complicated and the real type we alias to ceph:message (or whatever) is larger, but it doesn't impose load on everyday developers adding features to RADOS. 3) If we restrict ourselves to a single-core OSD, porting the code away from that will be nearly impossible. If we establish practices from the start which allow us to go multi-core, it's easy to switch to single-core with a few type aliases and get whatever speed benefits it promises us. -Greg
