I'm not sure if it will be in the initial pacific release but it will certainly get backported as soon as possible. On Fri, Feb 12, 2021 at 8:51 AM John Fulton <johfulto@xxxxxxxxxx> wrote: > > On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@xxxxxxxxxx> wrote: > > > > > > On 2/11/21 5:11 PM, Sage Weil wrote: > > > Hi everyone, > > > > > > One of the things that ceph-ansible does that cephadm does not is > > > automatically adjust the osd_memory_target based on the amount of > > > memory and number of OSDs on the host. The approach is pretty simple: > > > if the not_hci flag is set, then we take some factor (.7) * total > > > memory and divide by osd count, and set osd_memory_target accordingly. > > > > > > We'd like to bring this "hands off" behavior over to cephadm in some > > > form.. ideally one that is a bit more sophisticated. The currently > > > proposal in written up in this pad: > > > > > > https://pad.ceph.com/p/autotune_memory_target > > Thank you. This would be very useful for the TripleO project's > integration of cephadm [1] to deploy colocated OpenStack/Ceph. > > I understand you're still scoping the feature but can you say at this > point if you think it would be available in Pacific? > > Thanks, > John > > [1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ceph.html > > > > > The key ideas: > > > - An autotune_memory_target config option (boolean) can be enabled for > > > a specific daemon, class, host, or the whole cluster > > > - It will apply not just to OSDs but other daemons. Currently mon and > > > osd have a memory_target config; MDS has a similar setting for its > > > cache that we can work with. The other daemons don't have > > > configurable memory footprints (for the most part) but we can come up > > > with some estimate of their *usage* based on some conservative > > > estimates, maybe with cluster size factored in. > > > - The autotuner would look at other (ceph) daemons on the host that > > > aren't being tuned and deduct that memory from the total available, > > > and then divvy up the remaining memory among the autotuned daemons. > > > Initially we'd only support autotuning osd, mon, and mds. > > > > > > There are some open questions about how to implement some of the > > > details, though. > > > > > > For instance, the current proposal doesn't record what the "tuned" > > > memory is in ceph anywhere, but rather would look at what 'cephadm ls' > > > says about the deployed containers' limits, compare that to what it > > > thinks should happen, and redeploy daemons as needed. > > > > > > It might also make sense to allow these targets to be expressed via > > > the 'service spec' yaml, which is more or less equivalent to the CRs > > > in Rook (which currently do allow osd memory target to be set). > > > > > > I'm not entirely sure if this should be an orchestrator function > > > (i.e., work with both rook and cephadm) or cephadm-only. It would be > > > helpful to hear whether this sort of function would make any sense in > > > a rook cluster... are there cases where the kubernetes nodes are > > > dedicated to ceph and it makes sense to magically slurp up all of the > > > hosts' RAM? We probably don't want to be in the business of checking > > > with kubernetes about unused host resources and adjusting things > > > whenever the k8s scheduler does something on the host... > > > > > > sage > > > > > > One question is what you do when you want to add more daemons to a node > > after the targets have already been set. An idea I've kicked around for > > bare-metal clusters is to let the daemons float their memory target > > between lower and upper bounds based on how close to a global memory > > target all daemons on a particular node are in aggregate. Each daemon > > would be constantly re-evaluating it's local target like we already do > > when determining the aggregate cache budget based on that memory > > target. One really neat thing you could with this model is set the > > aggressiveness of each daemon based on the priorities and sizes of > > unfulfilled allocation requests in the priority cache manager. The idea > > being that daemons that really aggressively want memory at high > > priorities can float their targets up and daemons that don't really care > > much can float their targets down. Ultimately all daemons would still > > be governed based on that global target though. (ie if we get close to > > or over that global target, even aggressive daemons would necessarily > > back off and reduce their targets). I suspect this can be done on > > bare metal without strictly requiring coordination in the mgr (though we > > could certainly do it that way too). > > > > I agree for containers. It's...different. I don't know if we can even > > really adjust container resource limits on the fly yet. We could set the > > container limits really high (which is basically what we already do, the > > memory_target is already way lower than the container limit). I don't > > know that we want to be trying to coordinate memory usage behind the > > containers back like what I proposed above. Ideally there would be some > > kind of rich api where something inside the container can negotiate for > > more (or less) memory for the container as a whole. I don't think > > anything like this exists though. > > > > > > Mark > > > > > > > > > > _______________________________________________ > > > Dev mailing list -- dev@xxxxxxx > > > To unsubscribe send an email to dev-leave@xxxxxxx > > _______________________________________________ > > Dev mailing list -- dev@xxxxxxx > > To unsubscribe send an email to dev-leave@xxxxxxx > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
