Re: ceph_leadership_team_meeting_s18e06.mkv

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]



we have also experienced several ceph-mgr oom kills on ceph v16.2.13 on
120T/200T data.

Is there any tracker about the problem?

Does upgrade to 17.x "solves" the problem?

Kind regards,

On Wed, Sep 6, 2023 at 9:36 PM Ernesto Puerta <epuertat@xxxxxxxxxx> wrote:

> Dear Cephers,
> Today brought us an eventful CTL meeting: it looks like Jitsi recently
> started
> requiring user authentication
> <> (anonymous users
> will get a "Waiting for a moderator" modal), but authentication didn't work
> against Google or GitHub accounts, so we had to move to the good old Google
> Meet.
> As a result of this, Neha has kindly set up a new private Slack channel
> (#clt) to allow for quicker communication among CLT members (if you usually
> attend the CLT meeting and have not been added, please ping any CLT member
> to request that).
> Now, let's move on the important stuff:
> *The latest Pacific Release (v16.2.14)*
> *The Bad*
> The 14th drop of the Pacific release has landed with a few hiccups:
>    - Some .deb packages were made available to before
>    the release process completion. Although this is not the first time it
>    happens, we want to ensure this is the last one, so we'd like to gather
>    ideas to improve the release publishing process. Neha encouraged
> everyone
>    to share ideas here:
>       -
>       -
>       - v16.2.14 also hit issues during the ceph-container stage. Laura
>    wanted to raise awareness of its current setbacks
>    <> and collect ideas to tackle
>    them:
>       - Enforce reviews and mandatory CI checks
>       - Rework the current approach to use simple Dockerfiles
>       <>
>       - Call the Ceph community for help: ceph-container is currently
>       maintained part-time by a single contributor (Guillaume Abrioux).
> This
>       sub-project would benefit from the sound expertise on containers
> among Ceph
>       users. If you have ever considered contributing to Ceph, but felt a
> bit
>       intimidated by C++, Paxos and race conditions, ceph-container is a
> good
>       place to shed your fear.
> *The Good*
> Not everything about v16.2.14 was going to be bleak: David Orman brought us
> really good news. They tested v16.2.14 on a large production cluster
> (10gbit/s+ RGW and ~13PiB raw) and found that it solved a major issue
> affecting RGW in Pacific <>.
> *The Ugly*
> During that testing, they noticed that ceph-mgr was occasionally OOM killed
> (nothing new to 16.2.14, as it was previously reported). They already
> tried:
>    - Disabling modules (like the restful one, which was a suspect)
>    - Enabling debug 20
>    - Turning the pg autoscaler off
> Debugging will continue to characterize this issue:
>    - Enable profiling (Mark Nelson)
>    - Try Bloomberg's Python mem profiler
>    <> (Matthew Leonard)
> *Infrastructure*
> *Reminder: Infrastructure Meeting Tomorrow. **11:30-12:30 Central Time*
> Patrick brought up the following topics:
>    - Need to reduce the OVH spending ($72k/year, which is a good cut in the
>    Ceph Foundation budget, that's a lot less avocado sandwiches for the
> next
>    Cephalocon):
>       - Move services (e.g.: Chacra) to the Sepia lab
>       - Re-use CentOS (and any spared/unused) machines for devel purposes
>    - Current Ceph sys admins are overloaded, so devel/community involvement
>    would be much appreciated.
>    - More to be discussed in tomorrow's meeting. Please join if you
>    think you can help solve/improve the Ceph infrastrucru!
> *BTW*: today's CDM will be canceled, since no topics were proposed.
> Kind Regards,
> Ernesto
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux