Hi Reed,
That exactly matches what I'm seeing:
when iostat is working OK, I see ~5% CPU use by ceph-mgr
and when iostat freezes, ceph-mgr CPU increases to 100%
regarding OS, I'm using Scientific Linux 7.7
Kernel 3.10.0-957.21.3.el7.x86_64
I'm not sure if the mgr initiates scrubbing, but if so, this could be
the cause of the "HEALTH_WARN 20 pgs not deep-scrubbed in time" that we see.
Anyhow, many thanks for your input, please let me know if you have
further ideas :)
best,
Jake
On 8/27/19 2:01 PM, Reed Dier wrote:
> Curious what dist you're running on, as I've been having similar issues with instability in the mgr as well, curious if any similar threads to pull at.
>
> While the iostat command is running, is the active mgr using 100% CPU in top?
>
> Reed
>
>> On Aug 27, 2019, at 6:41 AM, Jake Grimmett <jog@xxxxxxxxxxxxxxxxx> wrote:
>>
>> Dear All,
>>
>> We have a new Nautilus (14.2.2) cluster, with 328 OSDs spread over 40 nodes.
>>
>> Unfortunately "ceph iostat" spends most of it's time frozen, with
>> occasional periods of working normally for less than a minute, then
>> freeze again for a couple of minutes, then come back to life, and so so
>> on...
>>
>> No errors are seen on screen, unless I press CTRL+C when iostat is stalled:
>>
>> [root@ceph-s3 ~]# ceph iostat
>> ^CInterrupted
>> Traceback (most recent call last):
>> File "/usr/bin/ceph", line 1263, in <module>
>> retval = main()
>> File "/usr/bin/ceph", line 1194, in main
>> verbose)
>> File "/usr/bin/ceph", line 619, in new_style_command
>> ret, outbuf, outs = do_command(parsed_args, target, cmdargs,
>> sigdict, inbuf, verbose)
>> File "/usr/bin/ceph", line 593, in do_command
>> return ret, '', ''
>> UnboundLocalError: local variable 'ret' referenced before assignment
>>
>> Observations:
>>
>> 1) This problem does not seem to be related to load on the cluster.
>>
>> 2) When iostat is stalled the dashboard is also non-responsive, if
>> iostat is working, the dashboard also works.
>>
>> Presumably the iostat and dashboard problems are due to the same
>> underlying fault? Perhaps a problem with the mgr?
>>
>>
>> 3) With iostat working, tailing /var/log/ceph/ceph-mgr.ceph-s3.log
>> shows:
>>
>> 2019-08-27 09:09:56.817 7f8149834700 0 log_channel(audit) log [DBG] :
>> from='client.4120202 -' entity='client.admin' cmd=[{"width": 95,
>> "prefix": "iostat", "poll": true, "target": ["mgr", ""], "print_header":
>> false}]: dispatch
>>
>> 4) When iostat isn't working, we see no obvious errors in the mgr log.
>>
>> 5) When the dashboard is not working, mgr log sometimes shows:
>>
>> 2019-08-27 09:18:18.810 7f813e533700 0 mgr[dashboard]
>> [::ffff:10.91.192.36:43606] [GET] [500] [2.724s] [jake] [1.6K]
>> /api/health/minimal
>> 2019-08-27 09:18:18.887 7f813e533700 0 mgr[dashboard] ['{"status": "500
>> Internal Server Error", "version": "3.2.2", "detail": "The server
>> encountered an unexpected condition which prevented it from fulfilling
>> the request.", "traceback": "Traceback (most recent call last):\\n File
>> \\"/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py\\", line 656,
>> in respond\\n response.body = self.handler()\\n File
>> \\"/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py\\", line
>> 188, in __call__\\n self.body = self.oldhandler(*args, **kwargs)\\n
>> File \\"/usr/lib/python2.7/site-packages/cherrypy/_cptools.py\\", line
>> 221, in wrap\\n return self.newhandler(innerfunc, *args, **kwargs)\\n
>> File \\"/usr/share/ceph/mgr/dashboard/services/exception.py\\", line
>> 88, in dashboard_exception_handler\\n return handler(*args,
>> **kwargs)\\n File
>> \\"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\\", line 34,
>> in __call__\\n return self.callable(*self.args, **self.kwargs)\\n
>> File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line
>> 649, in inner\\n ret = func(*args, **kwargs)\\n File
>> \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 192, in
>> minimal\\n return self.health_minimal.all_health()\\n File
>> \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 51, in
>> all_health\\n result[\'pools\'] = self.pools()\\n File
>> \\"/usr/share/ceph/mgr/dashboard/controllers/health.py\\", line 167, in
>> pools\\n pools = CephService.get_pool_list_with_stats()\\n File
>> \\"/usr/share/ceph/mgr/dashboard/services/ceph_service.py\\", line 124,
>> in get_pool_list_with_stats\\n \'series\': [i for i in
>> stat_series]\\nRuntimeError: deque mutated during iteration\\n"}']
>>
>>
>> 6) IPV6 is normally disabled on our machines at the kernel level, via
>> grubby --update-kernel=ALL --args="ipv6.disable=1"
>>
>> This was done as 'disabling ipv6' interfered with the dashboard (giving
>> "HEALTH_ERR Module 'dashboard' has failed: error('No socket could be
>> created',) we re-enabling ipv6 on the mgr nodes only to fix this.
>>
>>
>> Ideas...?
>>
>> Should ipv6 be enabled, even if not configured, on all ceph nodes?
>>
>> Any ideas on fixing this gratefully received!
>>
>> many thanks
>>
>> Jake
>>
>> --
>> MRC Laboratory of Molecular Biology
>> Francis Crick Avenue,
>> Cambridge CB2 0QH, UK.
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
