Thank you Adam! After "orch daemon redeploy", all works as expected. Tony ________________________________________ From: Adam King <adking@xxxxxxxxxx> Sent: March 24, 2022 11:50 AM To: Tony Liu Cc: ceph-users@xxxxxxx; dev@xxxxxxx Subject: Re: Re: logging with container Hmm, I'm assuming from "Setting "log_to_stderr" doesn't help" you've already tried all the steps in https://docs.ceph.com/en/latest/cephadm/operations/#disabling-logging-to-journald. That's meant to be the steps for stopping cluster logs from going to the container logs. From my personal testing, just setting the global config options made it work for all the daemons without needing to redeploy or set any of the values at runtime. I verified locally after setting log to file to true as well as the steps in the posted link new logs were getting put in /var/log/ceph/<fsid>/mon.host1 file but the journal had no new logs after when I changed the settings. Perhaps because you've modified the values directly at runtime for the daemons it isn't picking up the set config options as runtime changes override config options? It could be worth trying just redeploying the daemons after having all 6 of the relevant config options set properly. I'll also note that I have been using podman. Not sure if there is some major logging difference between podman and docker. Thanks, - Adam King On Thu, Mar 24, 2022 at 1:00 PM Tony Liu <tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx>> wrote: Any comments on this? Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx>> Sent: March 21, 2022 10:01 PM To: Adam King Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>; dev@xxxxxxx<mailto:dev@xxxxxxx> Subject: Re: logging with container Hi Adam, When I do "ceph tell mon.ceph-1 config set log_to_file true", I see the log file is created. That confirms that those options in command line can only be override by runtime config change. Could you check mon and mgr logging on your setup? Can we remove those options in command line and let logging to be controlled by cluster configuration or configuration file? Another issue is that, log keeps going to /var/lib/docker/containers/<id>/<id>-json.log, which keeps growing up and it's not under logrotate management. How can I stop logging to container stdout/stderr? Setting "log_to_stderr" doesn't help. Thanks! Tony ________________________________________ From: Tony Liu <tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx>> Sent: March 21, 2022 09:41 PM To: Adam King Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>; dev@xxxxxxx<mailto:dev@xxxxxxx> Subject: Re: logging with container Hi Adam, # ceph config get mgr log_to_file true # ceph config get mgr log_file /var/log/ceph/$cluster-$name.log # ceph config get osd log_to_file true # ceph config get osd log_file /var/log/ceph/$cluster-$name.log # ls /var/log/ceph/fa771070-a975-11ec-86c7-e4434be9cb2e/ ceph-osd.10.log ceph-osd.13.log ceph-osd.16.log ceph-osd.19.log ceph-osd.1.log ceph-osd.22.log ceph-osd.4.log ceph-osd.7.log ceph-volume.log # ceph version ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) "log_to_file" and "log_file" are set the same for mgr and osd, but why there is osd log only, but no mgr log? Thanks! Tony ________________________________________ From: Adam King <adking@xxxxxxxxxx<mailto:adking@xxxxxxxxxx>> Sent: March 21, 2022 08:26 AM To: Tony Liu Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>; dev@xxxxxxx<mailto:dev@xxxxxxx> Subject: Re: logging with container Hi Tony, Afaik those container flags just set the defaults and the config options override them. Setting the necessary flags (https://docs.ceph.com/en/latest/cephadm/operations/#logging-to-files) seemed to work for me. [ceph: root@vm-00 /]# ceph config get osd.0 log_to_file false [ceph: root@vm-00 /]# ceph config show osd.0 log_to_file false [ceph: root@vm-00 /]# ceph config set global log_to_file true [ceph: root@vm-00 /]# ceph config set global mon_cluster_log_to_file true [ceph: root@vm-00 /]# ceph config get osd.0 log_to_file true [ceph: root@vm-00 /]# ceph config show osd.0 log_to_file true [ceph: root@vm-00 /]# ceph version ceph version 16.2.7-601-g179a7bca (179a7bca8a84771b0dde09e26f7a2146a985df90) pacific (stable) [ceph: root@vm-00 /]# exit exit [root@vm-00 ~]# ls /var/log/ceph/413f7ec8-a91e-11ec-9b02-52540092b5a3/ ceph.audit.log ceph.cephadm.log ceph.log ceph-mgr.vm-00.ukcctb.log ceph-mon.vm-00.log ceph-osd.0.log ceph-osd.10.log ceph-osd.2.log ceph-osd.4.log ceph-osd.6.log ceph-osd.8.log ceph-volume.log On Mon, Mar 21, 2022 at 1:06 AM Tony Liu <tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx><mailto:tonyliu0592@xxxxxxxxxxx<mailto:tonyliu0592@xxxxxxxxxxx>>> wrote: Hi, After reading through doc, it's still not very clear to me how logging works with container. This is with Pacific v16.2 container. In OSD container, I see this. ``` /usr/bin/ceph-osd -n osd.16 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug ``` When check ceph configuration. ``` # ceph config get osd.16 log_file /var/log/ceph/$cluster-$name.log # ceph config get osd.16 log_to_file true # ceph config show osd.16 log_to_file false ``` Q1, what's the intention of those log settings in command line? It's high priority and overrides configuration in file and mon. Is there any option not doing that when deploy the container? Q2, since log_to_file is set to false by command line, why there is still loggings in log_file? The same for mgr and mon. What I want is to have everything in log file and minimize the stdout and stderr from container. Because log file is managed by logrotate, it unlikely blow up disk space. But stdout and stderr from container is stored in a single file, not managed by logrotate. It may grow up to huge file. Also, it's easier to check log file by vi than "podman logs". And log file is also collected and stored by ELK for central management. Any comments how I can achieve what I want? Runtime override may not be the best option, cause it's not persistent. Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx><mailto:ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx><mailto:ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |