Hi,
I don't have much to comment about logging, I feel you though. I just
wanted to point out that the details about the large omap object
should be in the (primary) OSD log, not in the MON log:
grep -i "large omap"
/var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.*
/var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.1.log:2024-08-09T11:21:23.943+0000 7ffb66d10700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes):
2070
Regards,
Eugen
Zitat von Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx>:
Hi,
I have a bunch of long-standing struggles with the way Ceph handles
logging and I cannot figured out how to solve them. These issues are
basically the following:
- The log config options are utterly confusing and very badly documented
- Mon file logs are spammed with DBG-level cluster logs, no matter
what I configure
- syslog/journald gets only part of the messages, but file logs must
be turned off due to the spam above
- "audit" channel logs cannot be configured at all
From this, my following needs and questions arise, perhaps you can help:
- I want to receive cluster and audit logs on the MONs with level
"warn" or at most "info"
- I want everything to go to journald
- Where to cluster logs go if "clog_to_monitors" is off?
- What's the difference between the "mon_cluster_log_*" and "clog_*"
settings?
- What the hell does "mon_cluster_log_to_syslog_facility" do and
what does "audit=local0" mean or do?
A very annoying symptom of the wonky logging config is that I cannot
debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the
cluster log for 'Large omap object found' for more details.", but I
cannot do that, because without enabling the file-logging flood
gates, I never receive the required cluster log info at the monitors
and there seems to be no other way to debug this than to grep the
cluster log (why??).
My current log config is the following:
global advanced
clog_to_monitors true
global basic
err_to_syslog true
global basic
log_to_file false
global basic
log_to_stderr false
global basic
log_to_syslog true
mon advanced
mon_cluster_log_file /dev/null
mon advanced
mon_cluster_log_to_file false
mon advanced
mon_cluster_log_to_stderr false
mon advanced
mon_cluster_log_to_syslog true
mon advanced
mon_cluster_log_to_syslog_facility daemon
mon advanced
mon_cluster_log_to_syslog_level warn
Any help solving this conundrum is much appreciated. Thanks!
Janek
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
