On 11/29/18 7:17 PM, Gregory Farnum wrote:
On Thu, Nov 29, 2018 at 1:46 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
Bufferlist is a common theme in Mark's weekly performance meetings, and
today's was no exception. Radoslaw highlighted the cost of atomic ref
counting associated with sharing buffers, and there was broad agreement
that the seastar osd's data path should avoid using bufferlist entirely
for that reason.
However, one choice that I made in the design of the seastar messenger
was to use Ceph's existing bufferlist-based Message types so that their
encode/decode wouldn't diverge between the seastar osd and its librados
clients and peers like ceph-mon/mgr.
In order to create a bufferlist-free data path, we'll need to find some
abstraction that allows Message types to define a serialization format
that can work both for bufferlist and for seastar's native buffer types
- whether that covers all Message types, or just some subset of
important ones like MOSDOp/Reply.
Any thoughts on what this could look like?
Casey
On Thu, Nov 29, 2018 at 2:32 PM Noah Watkins <nwatkins@xxxxxxxxxx> wrote:
Potentially stupid question... if there exists a buffer container that
can work in the new data path _without reference counting_, why not
create a bufferlist super class that doesn't do reference counting?
Or even just let seastar wrap its own buffers in a new buffer ptr type
and generate a new bufferlist when it passes that data into the
non-seastar-ized code?
-Greg
Yeah, that's essentially what the messenger does now - it reads
seastar's native buffers off the network, wraps them in a buffer::raw
type, and passes them as bufferlists to ::decode_message(). There's a
similar conversion from bufferlist to seastar::net::packet for the
send() side.
So the challenge is to find a way to decode messages from their native
buffers, avoiding conversions to bufferlist until we need to pass them
to common code that expects bufferlist. Ideally, the data path won't
require any of these conversions.
