Re: Ceph Logging Configuration and "Large omap objects found"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I forgot to add this one to get the info from any admin node:

ceph log last 10 warn cluster
2024-08-09T11:21:23.949916+0000 osd.1 (osd.1) 6 : cluster [WRN] Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes): 2070 2024-08-09T11:21:27.723959+0000 mon.soc9-ceph (mon.0) 11905 : cluster [WRN] Health check failed: 1 large omap objects (LARGE_OMAP_OBJECTS)

Then you don't have to search each node for logs.

Zitat von Eugen Block <eblock@xxxxxx>:

Hi,

I don't have much to comment about logging, I feel you though. I just wanted to point out that the details about the large omap object should be in the (primary) OSD log, not in the MON log:

grep -i "large omap" /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.* /var/log/ceph/bce93c48-5552-11ef-8ba9-fa163e2ad8c5/ceph-osd.1.log:2024-08-09T11:21:23.943+0000 7ffb66d10700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 3:592df674:::file:head PG: 3.2e6fb49a (3.0) Key count: 363 Size (bytes): 2070

Regards,
Eugen

Zitat von Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx>:

Hi,

I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following:

- The log config options are utterly confusing and very badly documented
- Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above
- "audit" channel logs cannot be configured at all

From this, my following needs and questions arise, perhaps you can help:

- I want to receive cluster and audit logs on the MONs with level "warn" or at most "info"
- I want everything to go to journald
- Where to cluster logs go if "clog_to_monitors" is off?
- What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do?

A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??).

My current log config is the following:

global                    advanced clog_to_monitors                                     true global                    basic err_to_syslog                                        true global                    basic log_to_file                                          false global                    basic log_to_stderr                                        false global                    basic log_to_syslog                                        true mon                       advanced mon_cluster_log_file                                 /dev/null mon                       advanced mon_cluster_log_to_file                              false mon                       advanced mon_cluster_log_to_stderr                            false mon                       advanced mon_cluster_log_to_syslog                            true mon                       advanced mon_cluster_log_to_syslog_facility                   daemon mon                       advanced mon_cluster_log_to_syslog_level                      warn


Any help solving this conundrum is much appreciated. Thanks!

Janek


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux