i am trying to note down the gist of the discussion with Radoslaw over #ceph-devel today. we were talking about the movable connection as one of the requirements of m:n mapping. m:n mapping requires us to distribute the connections to different cores in a predicable ways. * for POSIX stack: currently, seastar listens on a single thread, and then distribute the established connection to different cores in round-robin manner, see posix_server_socket_impl<Transport>::accept() in posix-stack.cc * for native stack: seastar just handle the established connection on the core where it is established, see native_server_socket_impl<Protocol>::accept() in native-stack-impl.hh. to allow sharing the connections. we need have a predicable connection placement policy, apparently, the plain round-robin can not fulfill this requirement. so we have 3 options. a) note down where the connection is established, and always perform i/o on that core. and handle the complexity of connection replacement across cores. b) change seastar, so that it allows us to plug-in a placement policy class to customize the placement. this change should apply to both POSIX and native stacks if we want to make the API consistent between these two stack implementation. c) change seastar, so that it allows us to move an established connection across core. one option is to switch from lw_shared_ptr<pollable_fd> to foreign_ptr<lw_shared_ptr<pollable_fd>> in posix_connected_socket_impl and its friends @Yingxin, also as a side note, if we always want to re-distribute the connections to different cores. there is no need to use reuseport and to listen on multiple cores concurrently. On Mon, Jan 7, 2019 at 10:29 AM Radoslaw Zarzynski <rzarzyns@xxxxxxxxxx> wrote: > > On Sat, Jan 5, 2019 at 3:42 PM kefu chai <tchaikov@xxxxxxxxx> wrote: > > > recently, when we are working on cross-core messenger[0], we found > > that, in order to share the connection between cores we need to have > > types like "seastar::lw_shared_ptr<seastar::foreign_ptr<ConnectionRef>>", > > because > > - the connections to peer OSDs are shared across cores, > > - the connections are shared by multiple continuations on the local > > core -- either locally or remotely. > > Well, I perceive this type as a clever solution to get back some of > the properties seastar::foreign_ptr had taken from us. What worries > me is that we need such heavy artillery at the very early stage of > project life. > > > and we need to perform i/o on the core where the connection is > > established. > > Many problems can be resolved by introducing another indirection > layer. This becomes harder if a pointer that should be wrapped with > foreign_ptr resides in a third-party component. > > Regards, > Radek -- Regards Kefu Chai
