[adding ceph-devel]
On 01/30/2018 01:56 PM, Casey Bodley wrote:
Hey Josh,
I heard you mention in the call yesterday that you're looking into this
part of seastar integration. I was just reading through the relevant
code over the weekend, and wanted to compare notes:
in seastar, all cross-core communication goes through lockfree spsc
queues, which are encapsulated by 'class smp_message_queue' in
core/reactor.hh. all of these queues (smp::_qs) are allocated on startup
in smp::configure(). early in reactor::run() (which is effectively each
seastar thread's entrypoint), it registers a smp_poller to poll all of
the queues directed at that cpu
what we need is a way to inject messages into each seastar reactor from
arbitrary/external threads. our requirements are very similar to
smp_message_queue's, with a few exceptions:
-each seastar reactor should be the single consumer of a multi-producer
queue, and poll on that as well
Yes, this is what I was thinking too, maybe a boost::lockfree::queue
-the submit() function would return void instead of a future (which
removes the need for a lot of other stuff, like the _completions queue,
async_work_item::_promise, etc)
figuring out how to factor this stuff out of smp_message_queue cleanly
is the hard part i guess
I was thinking it could start off as a separate implementation, but
hadn't looked too closely at sharing pieces of it.
in terms of startup, it could be allocated as a static array similar to
smp::_qs (except it would be dimensioned by [smp::count] instead of
[smp::count][smp::count]). then a new function could be added alongside
smp::submit_to() that submits to the given cpu's external queue (and
also returns void). this stuff should probably be disabled by default,
and only turned on if enabled in configuration
++
for a super simple unit test, you could spawn an external thread that
does something like this:
std::mutex mutex;
std::condition_variable cond;
std::atomic<int> completions = 0;
// submit a message to each reactor
for (int i = 0; i < smp::count; i++) {
smp::external_submit_to(i, [&] { ++completions; cond.notify_one(); });
}
// wait for all completions
std::unique_lock lock(mutex);
cond.wait(lock, [&] { return completions == smp::count; });
Yeah, this looks like a nice example.
Sorry that I've been slow to help with this - keep me posted?
No worries, I've been slow about this too - I've asked Kefu to look at
it this morning, so I'm sure he'll have some more thoughts soon.
Josh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
