Re: ceph-mgr high CPU utilization

Andras Pataki <apataki@xxxxxxxxxxxxxxxxxxxxx> · Thu, 7 May 2020 21:55:25 -0400

Here it is: https://tracker.ceph.com/issues/45439
The balancer problems we saw might be a consequence of the mgr falling 
behind, since even after disabling the balancer, the mgr stays 
overloaded.  I ended up removing all upmaps and leaving the balancer 
disabled since it further increases the osd map size (we had about 2900 
upmaps).

Andras

On 5/7/20 6:42 PM, Brad Hubbard wrote:
Could you create a tracker for this and attach an osdmap as well as
some recent balancer output (perhaps at a higher debug level if
possible)?

There are some improvements awaiting backport to nautilus for the
C++/python interface just FYI [0]

You might also look at gathering output using something like [1] to
try to narrow down further what is causing the high CPU consumption.

[0] https://github.com/ceph/ceph/pull/34356
[1] https://github.com/markhpc/gdbpmp

On Fri, May 8, 2020 at 1:10 AM Andras Pataki
<apataki@xxxxxxxxxxxxxxxxxxxxx> wrote:
Hi everyone,

After some investigation, it looks like on our large cluster, ceph-mgr
is not able to keep up with the status updates from about 3500 OSDs.  By
default OSDs send updates to ceph-mgr every 5 seconds, which, in our
case, turns to about 700 messages/s to ceph-mgr.  It looks from gdb
traces that ceph-mgr runs some python code for each of them - so 700
python snipets/s might be too much. Increasing mgr_stats_period to 15
seconds reduces the load and brings ceph-mgr back to responsive again.
Unfortunately this isn't sustainable since if we were to expand the
cluster, we'd need to further reduce the update frequency from OSDs.

I also checked our other clusters and they have about proportionately
lower load on ceph-mgr based on their OSD counts.

Any thoughts about the scalability of ceph-mgr to a large number of
OSDs?  We recently upgraded this cluster from Mimic, where we didn't see
this issue.

Andras

On 5/1/20 8:48 AM, Andras Pataki wrote:
Also just a follow-up on the misbehavior of ceph-mgr.  It looks like
the upmap balancer is not acting reasonably either.  It is trying to
create upmap entries every minute or so - and claims to be successful,
but they never show up in the OSD map.  Setting the logging to
'debug', I see upmap entries created such as:

2020-05-01 08:43:07.909 7fffca074700  4 mgr[balancer] ceph osd
pg-upmap-items 9.60c4 mappings [{'to': 3313L, 'from': 3371L}]
2020-05-01 08:43:07.909 7fffca074700  4 mgr[balancer] ceph osd
pg-upmap-items 9.632b mappings [{'to': 2187L, 'from': 1477L}]
2020-05-01 08:43:07.909 7fffca074700  4 mgr[balancer] ceph osd
pg-upmap-items 9.6b9c mappings [{'to': 3315L, 'from': 3371L}]
2020-05-01 08:43:07.909 7fffca074700  4 mgr[balancer] ceph osd
pg-upmap-items 9.6bf6 mappings [{'to': 1581L, 'from': 1477L}]
2020-05-01 08:43:07.909 7fffca074700  4 mgr[balancer] ceph osd
pg-upmap-items 9.7da4 mappings [{'to': 2419L, 'from': 2537L}]
...
2020-05-01 08:43:07.909 7fffca074700 20 mgr[balancer] commands
[<mgr_module.CommandResult object at 0x7fffcc990550>,
<mgr_module.CommandResult object at 0x7fffcc990fd0>,
<mgr_module.CommandResult object at 0x7fffcc9907d0>, <mgr_module.Com
mandResult object at 0x7fffcc990650>, <mgr_module.CommandResult object
at 0x7fffcc990610>, <mgr_module.CommandResult object at
0x7fffcc990f50>, <mgr_module.CommandResult object at 0x7fffcc990bd0>,
<mgr_module.CommandResult object at 0x7ff
fcc990d90>, <mgr_module.CommandResult object at 0x7fffcc990ad0>,
<mgr_module.CommandResult object at 0x7fffcc990410>,
<mgr_module.CommandResult object at 0x7fffbed241d0>,
<mgr_module.CommandResult object at 0x7fff6a6caf90>, <mgr_module.Co
mmandResult object at 0x7fffbed242d0>, <mgr_module.CommandResult
object at 0x7fffbed24d90>, <mgr_module.CommandResult object at
0x7fffbed24d50>, <mgr_module.CommandResult object at 0x7fffbed24550>,
<mgr_module.CommandResult object at 0x7f
ffbed245d0>, <mgr_module.CommandResult object at 0x7fffbed24510>,
<mgr_module.CommandResult object at 0x7fffbed24690>,
<mgr_module.CommandResult object at 0x7fffbed24990>]
...
2020-05-01 08:43:16.733 7fffca074700 20 mgr[balancer] done
...

but these mappings do not show up in the osd dump.  And a minute
later, the balancer tries again and comes up with a set of very
similar mappings (same from and to OSDs, slightly different PG
numbers) - and keeps going like that every minute without any progress
(the set of upmap entries stays the same, does not increase).

Andras

On 5/1/20 8:12 AM, Andras Pataki wrote:
I'm wondering if anyone still sees issues with ceph-mgr using CPU and
being unresponsive even in recent Nautilus releases.  We upgraded our
largest cluster from Mimic to Nautilus (14.2.8) recently - it has
about 3500 OSDs.  Now ceph-mgr is constantly at 100-200% CPU (1-2
cores), and becomes unresponsive after a few minutes.  The
finisher-Mgr queue length grows (I've seen it at over 100k) - similar
symptoms as seen with earlier Nautilus releases by many. This is what
it looks like after an hour of running:

     "finisher-Mgr": {
         "queue_len": 66078,
         "complete_latency": {
             "avgcount": 21,
             "sum": 2098.408767721,
             "avgtime": 99.924227034
         }
     },

We have a pretty vanilla manager config, only the balancer is enabled
in upmap mode.  Here are the enabled modules:

     "always_on_modules": [
         "balancer",
         "crash",
         "devicehealth",
         "orchestrator_cli",
         "progress",
         "rbd_support",
         "status",
         "volumes"
     ],
     "enabled_modules": [
         "restful"
     ],

Any ideas or outstanding issues in this area?

Andras

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx