Hi Zakhar,
I use this docker config file (/etc/docker/daemon.json) to limit its log size:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}
I only changed the default rocksdb log level with
ceph config set global debug_rocksdb 1/5
And ceph has a lot of log level configs (named like debug_*) that you can adjust.
The logs you pasted are cluster logs. And it seems we don’t have a “mon_cluster_log_to_stderr_level” config. Maybe you can turn it off by changing “mon_cluster_log_to_stderr” to false. But I don’t recommend that unless you already store the cluster logs somewhere else (e.g. mon_cluster_log_to_file).
Weiwen Hu
发件人: Zakhar Kirpichenko<mailto:zakhar@xxxxxxxxx>
发送时间: 2022年1月27日 17:19
收件人: ceph-users<mailto:ceph-users@xxxxxxx>
主题: Ceph 16.2.7 + cephadm, how to reduce logging and trim existing logs?
Hi!
I have a Ceph cluster version 16.2.7 built with cephadm. Ceph logs in
/var/log/ceph/ all rotate nicely, but container logs in
/var/lib/docker/containers/ keep growing. For example, a log of a monitor
container has reached almost 3 GB since December 2021:
09:18 [root@ceph02
/var/lib/docker/containers/04807066d68dab96c506e03c2a1e2d3d42bde6677353db5728c5a2ed637579fe]#
ls -al *log
-rw-r----- 1 root root 3011531294 Jan 27 09:18
04807066d68dab96c506e03c2a1e2d3d42bde6677353db5728c5a2ed637579fe-json.log
The log is full of debug info, such as
{"log":"cluster 2022-01-27T09:13:47.026820+0000 mgr.ceph01.vankui
(mgr.55010800) 2575830 : cluster [DBG] pgmap v2156823: 1824 pgs: 1824
active+clean; 4.4 TiB data, 28 TiB used, 549 TiB / 577 TiB avail; 2.6 MiB/s
rd, 7.0 MiB/s wr, 915
op/s\n","stream":"stderr","time":"2022-01-27T09:13:48.274941081Z"}
{"log":"audit 2022-01-27T09:13:58.659145+0000 mon.ceph04 (mon.3) 427114 :
audit [DBG] from='client.? a.b.c.d:0/1284725261' entity='client.cinder'
cmd=[{\"prefix\":\"osd pool get-quota\", \"pool\": \"volumes\",
\"format\":\"json\"}]:
dispatch\n","stream":"stderr","time":"2022-01-27T09:13:59.26827113Z"}
Other containers, including managers and OSDs, also appear to log excessive
debug information. What is the best way to reduce logging of all containers
to reasonable levels and trim the existing container logs?
I would very much appreciate any advice.
Best regards,
Zakhar
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
