I forgot to add this one to get the info from any admin node:
ceph log last 10 warn cluster
2024-08-09T11:21:23.949916+0000 osd.1 (osd.1) 6 : cluster [WRN] Large
omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0)
Key count: 363 Size (bytes): 2070
2024-08-09T11:21:27.723959+0000 mon.soc9-ceph (mon.0) 11905 : cluster
[WRN] Health check failed: 1 large omap objects (LARGE_OMAP_OBJECTS)
Then you don't have to search each node for logs.
Zitat von Eugen Block <eblock@xxxxxx>:
Hi,
I don't have much to comment about logging, I feel you though. I
just wanted to point out that the details about the large omap
object should be in the (primary) OSD log, not in the MON log:
grep -i "large omap"
/var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.*
/var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.1.log:2024-08-09T11:21:23.943+0000 7ffb66d10700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes):
2070
Regards,
Eugen
Zitat von Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx>:
Hi,
I have a bunch of long-standing struggles with the way Ceph handles
logging and I cannot figured out how to solve them. These issues
are basically the following:
- The log config options are utterly confusing and very badly documented
- Mon file logs are spammed with DBG-level cluster logs, no matter
what I configure
- syslog/journald gets only part of the messages, but file logs
must be turned off due to the spam above
- "audit" channel logs cannot be configured at all
From this, my following needs and questions arise, perhaps you can help:
- I want to receive cluster and audit logs on the MONs with level
"warn" or at most "info"
- I want everything to go to journald
- Where to cluster logs go if "clog_to_monitors" is off?
- What's the difference between the "mon_cluster_log_*" and
"clog_*" settings?
- What the hell does "mon_cluster_log_to_syslog_facility" do and
what does "audit=local0" mean or do?
A very annoying symptom of the wonky logging config is that I
cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says
"Search the cluster log for 'Large omap object found' for more
details.", but I cannot do that, because without enabling the
file-logging flood gates, I never receive the required cluster log
info at the monitors and there seems to be no other way to debug
this than to grep the cluster log (why??).
My current log config is the following:
global advanced
clog_to_monitors true
global basic
err_to_syslog true
global basic
log_to_file false
global basic
log_to_stderr false
global basic
log_to_syslog true
mon advanced
mon_cluster_log_file /dev/null
mon advanced
mon_cluster_log_to_file false
mon advanced
mon_cluster_log_to_stderr false
mon advanced
mon_cluster_log_to_syslog true
mon advanced
mon_cluster_log_to_syslog_facility daemon
mon advanced
mon_cluster_log_to_syslog_level warn
Any help solving this conundrum is much appreciated. Thanks!
Janek
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
