On Thu, Mar 7, 2019 at 7:26 AM Sage Weil <sweil@xxxxxxxxxx> wrote:
>
> On Thu, 7 Mar 2019, Sage Weil wrote:
> > On Thu, 7 Mar 2019, Sebastien Han wrote:
> > > Let me take back something, afterthoughts, yes "memory.request" is
> > > what we are interested in, the only thing is that there is no
> > > /sys/fs/cgroup/memory equivalent.
> >
> > Okay, if that's the case, then we have to take the limit from the
> > kubernetes-specific environment variable. The downside is it's
> > kube-specific, but so be it.
> >
> > > If we take the OSD example, we could set REQUEST to 1GB and LIMIT to
> > > 2GB (making up numbers again) which leaves 1GB for backfilling and
> > > other stuff.
> >
> > Just a side-note here: there is no longer an recovery vs non-recovery
> > memory utilization delta. IIRC we even backported that config
> > osd_pg_log_{min,max}_entries change to luminous too.
> >
> > > The last thing is, as part of these 4GB how much do we want to give to
> > > the osd_memory_target.
> >
> > I think we should set osd_memory_target to POD_MEMORY_REQUEST if it's
> > non-zero. If it is 0 and POD_MEMORY_LIMIT is set, I suggest .8 * that.
>
> Hmm, I think the place to do that is here:
>
> https://github.com/ceph/ceph/blob/master/src/common/config.cc#L456
>
> It's a bit awkward to parse the env variable into one of
> {osd,mds,mon}_memory_target based on the daemon type, but it could be done
> (once the others exist).
>
> I wonder if, instead, we should have named the option just memory_target.
> :/
The MDS part of this is tracked here: http://tracker.ceph.com/issues/36663
I've asked +Sidharth to start working on an intermediate solution of
the MDS of setting mds_cache_memory_limit based on the value of
mds_memory_target.
However, I also like the idea of memory_target. We can add code in
global init that configures memory_target based on the environment
variables according to the formula you outlined above. A thread
watching /sys/fs/cgroup/memory/memory.limit_in_bytes for changes may
make sense too?
--
Patrick Donnelly
