On Fri, Aug 24, 2018 at 3:04 AM, kefu chai <tchaikov@xxxxxxxxx> wrote: > this is a summary of discussion on osd-seastar we had in a meeting in this week. > > seastar use share-nothing design to take the advantage of multi-core > hardware. but there are some inherent problems in OSD. in seastar-osd, > we will have a sharded osd service listening on given port on all > configured cores in parallel using SO_REUSEPORT, so the connections > are evenly distributed [0] across all seastar reactors. > > also, in seastar-osd, to shard PGs on different cores looks like an > intuitive design. for instance, we can > - ensure the order of osd op to maintain a pglog > - have better control of the io queue-depth of the storage device > - maintain a consistent state without extra "locking" of the > underlying ObjectStore and PG instances. > > but we cannot enforce a client to send requests to a single PG, or the > PGs which happen to be hosted by the core which accepts the connection > from this client. so i think we can only have a run-to-completion > session for a request chain which is targeting a certain PG, and > forward the client to whichever the PG it wants to talk to. this > cross-core communication is inevitable, i think. Yep, this is certainly the model I've always expected — the crossbar is inevitable, but hopefully it can be one time on the way in and one time to send the reply back. Figuring out details like whose memory we allocate for sending replies and things may be interesting to explore when we get there, but that's not a problem for today. -Greg > > to avoid starving low traffic connection by high traffic client on a > certain core, we use the `Throttle` attached to each connection. see > SocketConnection::maybe_throttle(). > > --- > > [0] https://lwn.net/Articles/542629/ > -- > Regards > Kefu Chai
