Hmm, I'm assuming from "Setting "log_to_stderr" doesn't help" you've already tried all the steps in https://docs.ceph.com/en/latest/cephadm/operations/#disabling-logging-to-journald. That's meant to be the steps for stopping cluster logs from going to the container logs. From my personal testing, just setting the global config options made it work for all the daemons without needing to redeploy or set any of the values at runtime. I verified locally after setting log to file to true as well as the steps in the posted link new logs were getting put in /var/log/ceph/<fsid>/mon.host1 file but the journal had no new logs after when I changed the settings. Perhaps because you've modified the values directly at runtime for the daemons it isn't picking up the set config options as runtime changes override config options? It could be worth trying just redeploying the daemons after having all 6 of the relevant config options set properly. I'll also note that I have been using podman. Not sure if there is some major logging difference between podman and docker.
Thanks,
- Adam King
On Thu, Mar 24, 2022 at 1:00 PM Tony Liu <tonyliu0592@xxxxxxxxxxx> wrote:
Any comments on this?
Thanks!
Tony
________________________________________
From: Tony Liu <tonyliu0592@xxxxxxxxxxx>
Sent: March 21, 2022 10:01 PM
To: Adam King
Cc: ceph-users@xxxxxxx; dev@xxxxxxx
Subject: [ceph-users] Re: logging with container
Hi Adam,
When I do "ceph tell mon.ceph-1 config set log_to_file true",
I see the log file is created. That confirms that those options in command line
can only be override by runtime config change.
Could you check mon and mgr logging on your setup?
Can we remove those options in command line and let logging to be controlled
by cluster configuration or configuration file?
Another issue is that, log keeps going to /var/lib/docker/containers/<id>/<id>-json.log,
which keeps growing up and it's not under logrotate management. How can I stop
logging to container stdout/stderr? Setting "log_to_stderr" doesn't help.
Thanks!
Tony
________________________________________
From: Tony Liu <tonyliu0592@xxxxxxxxxxx>
Sent: March 21, 2022 09:41 PM
To: Adam King
Cc: ceph-users@xxxxxxx; dev@xxxxxxx
Subject: [ceph-users] Re: logging with container
Hi Adam,
# ceph config get mgr log_to_file
true
# ceph config get mgr log_file
/var/log/ceph/$cluster-$name.log
# ceph config get osd log_to_file
true
# ceph config get osd log_file
/var/log/ceph/$cluster-$name.log
# ls /var/log/ceph/fa771070-a975-11ec-86c7-e4434be9cb2e/
ceph-osd.10.log ceph-osd.13.log ceph-osd.16.log ceph-osd.19.log ceph-osd.1.log ceph-osd.22.log ceph-osd.4.log ceph-osd.7.log ceph-volume.log
# ceph version
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
"log_to_file" and "log_file" are set the same for mgr and osd, but why there is osd log only,
but no mgr log?
Thanks!
Tony
________________________________________
From: Adam King <adking@xxxxxxxxxx>
Sent: March 21, 2022 08:26 AM
To: Tony Liu
Cc: ceph-users@xxxxxxx; dev@xxxxxxx
Subject: Re: [ceph-users] logging with container
Hi Tony,
Afaik those container flags just set the defaults and the config options override them. Setting the necessary flags (https://docs.ceph.com/en/latest/cephadm/operations/#logging-to-files) seemed to work for me.
[ceph: root@vm-00 /]# ceph config get osd.0 log_to_file
false
[ceph: root@vm-00 /]# ceph config show osd.0 log_to_file
false
[ceph: root@vm-00 /]# ceph config set global log_to_file true
[ceph: root@vm-00 /]# ceph config set global mon_cluster_log_to_file true
[ceph: root@vm-00 /]# ceph config get osd.0 log_to_file
true
[ceph: root@vm-00 /]# ceph config show osd.0 log_to_file
true
[ceph: root@vm-00 /]# ceph version
ceph version 16.2.7-601-g179a7bca (179a7bca8a84771b0dde09e26f7a2146a985df90) pacific (stable)
[ceph: root@vm-00 /]# exit
exit
[root@vm-00 ~]# ls /var/log/ceph/413f7ec8-a91e-11ec-9b02-52540092b5a3/
ceph.audit.log ceph.cephadm.log ceph.log ceph-mgr.vm-00.ukcctb.log ceph-mon.vm-00.log ceph-osd.0.log ceph-osd.10.log ceph-osd.2.log ceph-osd.4.log ceph-osd.6.log ceph-osd.8.log ceph-volume.log
On Mon, Mar 21, 2022 at 1:06 AM Tony Liu <tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx>> wrote:
Hi,
After reading through doc, it's still not very clear to me how logging works with container.
This is with Pacific v16.2 container.
In OSD container, I see this.
```
/usr/bin/ceph-osd -n osd.16 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug
```
When check ceph configuration.
```
# ceph config get osd.16 log_file
/var/log/ceph/$cluster-$name.log
# ceph config get osd.16 log_to_file
true
# ceph config show osd.16 log_to_file
false
```
Q1, what's the intention of those log settings in command line? It's high priority and overrides
configuration in file and mon. Is there any option not doing that when deploy the container?
Q2, since log_to_file is set to false by command line, why there is still loggings in log_file?
The same for mgr and mon.
What I want is to have everything in log file and minimize the stdout and stderr from container.
Because log file is managed by logrotate, it unlikely blow up disk space. But stdout and stderr
from container is stored in a single file, not managed by logrotate. It may grow up to huge file.
Also, it's easier to check log file by vi than "podman logs". And log file is also collected and
stored by ELK for central management.
Any comments how I can achieve what I want?
Runtime override may not be the best option, cause it's not persistent.
Thanks!
Tony
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
