On 1/5/19 8:41 AM, kefu chai wrote:
as you might know, seastar encourage a share-nothing programming
paradigm. as in previous discussions we found that there are always
some cross-core communications in the sharded seastar-osd, because
there are couple infrastructures could be shared by a sharded OSD,
namely:
- osdmap cache
- connection to peer OSDs, and heartbeats with them
- connection to monitor and mgr, and beacon/reports to them
- i/o to the underlying objectstore
recently, when we are working on cross-core messenger[0], we found
that, in order to share the connection between cores we need to have
types like "seastar::lw_shared_ptr<seastar::foreign_ptr<ConnectionRef>>",
because
- the connections to peer OSDs are shared across cores,
- the connections are shared by multiple continuations on the local
core -- either locally or remotely.
and we need to perform i/o on the core where the connection is
established. personally, i feel that it's a bad smell, as it's
complicated and always involves cross-core communications.
Radoslaw suggested an alternative: single-threaded OSD which pushes
the share-nothing design to another level. in this design, just like
the existing model, an OSD host will still have multiple instances of
OSD, but each instance of OSD will be running on and only on its own
designated core. nothing will be shared across these OSD instances. so
we can still benefit from Seastar, and at the same time, won't be
worried about the complexities and performance degradation due to
cross-core communications. this design resembles the co-located OSD
design we were talking about in the sense that all OSDs will reside in
the same process. but they are different in that, it enforces strict
share-nothing model.
but on the other side, single-threaded OSD has following
restrictions/assumptions:
- 1-to-1 mapping from core to OSD. some of the following questions
also apply to the NIC.
* Mark worried that what if we have more stores than cores. or more
cores than disks? how can we do the mapping. probably to group disks
into an LVM? but that would increase the load of the core which gets
mapped to that LVM volume. which causes imbalance of the load, i
think.
* how about more cores than stores?
* how to shard a high throughput storage device?
for instance, to take the full advantage of an NVMe storage
device, we might need to drive it with 4 or more cores. but how to do
it? can we leverage virtualization techniques like SPDK-vhost,
SPDK-Blobstore? for a device supporting SR-IOV, it'd be probably
simpler.
- unable to share the osdmap cache. considering a high-density storage
deployment, where more than 40 disks are squeezed into a host, if we
are not able to reuse the osdmap cache. that's a shame..
- unable to share the connection to peer OSDs, mon and mgr.
probably it's not a big deal in comparison to existing non
co-located OSD, but if we compare it with the co-located OSD, well,
you'll see what we will be missing.
we had some discussions on this topic recently on the crimson standup
and on the perf meeting. but i feel that the only consensus we reached
is that it's difficult to tell which way to go -- 1-1 mapping or m-n
mapping.
what i can think of is to avoid making the decision now, and instead
to encapsulate the difference between these two approach as much as
possible in smaller scopes. for instance, to hide the difference
between a shared messenger and a non-shared messenger in the
messenger's implementation itself, and provide a consistent API to the
caller/dispatcher. so we can switch over to the single-threaded OSD in
future if necessary with less pain. but i admit that it does not
address the complexity of the pain of
"seastar::lw_shared_ptr<seastar::foreign_ptr<ConnectionRef>>". =)
thoughts?
Hi Kefu,
I think you hit the nail on the head regarding the technical discussion
here. After the perf call last week I spent some time researching
SR-IOV and also reached out to some of the hardware vendors for
clarification regarding capabilities/plans regarding storage hardware
virtualization. So far my impression is:
- SR-IOV is fairly well supported on the networking side.
- Currently very few storage products support hardware virtualization
via multiple namespaces or SR-IOV VFs.
- The NVMe 1.3 spec will help, but I can't find any documentation
regarding enforcing minimum standards
- SR-IOV in general will require CPU/MB support in addition to
network/disk support.
- SR-IOV is probably the future, but we may not strictly need it for the
single-core OSD model (blobstore? blobfs? Direct access to HW queues?)
Perhaps folks from HW vendors can jump in and fix any misconceptions
here or provide guidance regarding future direction. I think a lot of
this comes down to how we view what an OSD actually represents. Is it
just some simple/dumb entity (almost like a shard) that executes on some
fraction of hardware that is governed at a higher level? Alternately,
does the OSD represent a grouping of hardware entities that have some
relationship and "closeness" to one another that can be exploited in
ways that we can't exploit at a higher level? How much autonomy does an
OSD have to make decisions on it's own? How should a user think about
OSDs that exist in this kind of model?
The single-core OSD approach is attractive in a lot of ways. You can
avoid the cross-core messenger issue. Avoid issues which historically
have required locking (pg!) and/or atomics. It also lends itself well
to concurrent OSDs running on the same filesystem or block devices, or,
in the future, using SR-IOV with SPDK for HW virtualization without any
kernel involvement. If we go this route, I think we should focus on
keeping the OSD as simple as possible as that's sort of the main
advantage with this approach. We'll really need to focus on supporting
massive numbers of OSDs smoothly and easily. We'll be relying heavily
on crush and the monitors to scale well. I imagine it could be very
successful for small-medium sized installations. Other questions: How
much memory does this waste? (Greg specifically pointed this out) What
do you do when the disk/core mapping isn't clean (increasing the number
of OSDs to get better balance exacerbates some of the issues)? What
does the PG mapping and pglog look like if you potentially have 8+
OSDs/NVMe drive? What is the emergent behavior of the system when you
take this model to the extreme?
An alternate I've been thinking about is to treat the OSD as a logical
grouping of close hardware (maybe a numa-node). Fewer OSDs and so a
lower requirement on the monitors and crush, but with increased OSD
complexity. Can we still shard internally and just eat memory copies
where appropriate (or deal with atomics and/or locking)? Does the
advent of nvdimm and other very fast persistent memory make crush less
relevant within a specific numa node for data placement (how valuable is
dynamic data placement within a numa node?). How much memory do we save
by avoiding have tons of tiny OSDs? My own goal with this model would
be to make the OSD as self-configuring as possible and avoid external
API complexity and decision making. The OSD takes control over a
specific hardware group and has far less external instructions regarding
how to make use of it. The goal would be to ask much less of ceph-disk,
ceph-volume, and ceph-ansible with the assumption that the OSD itself
can make better decisions about how to make use of resources.
I oscillate between both approaches, but I think we probably want to
pick one or the other in the long run. I'm afraid if we try to support
both models long term we'll end up with few advantages and most of the
disadvantages of both. On the other hand, it might make sense to
prototype both approaches (maybe without any kind of recovery features
or other complexity that would slow development down) just to get a
sense of what each feels like.
Mark
---
[0] https://github.com/ceph/ceph/pull/24945
