Hi Ben, After chown tp 472, “systemctl daemon-reload” changes it back to 167. I also notice that these are still from docker.io while the rest are from quay /home/general# docker ps --no-trunc | grep docker 93b8c3aa33580fb6f4951849a6ff9c2e66270eb913b8579aca58371ef41f2d6c docker.io/grafana/grafana:6.7.4 "/run.sh" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 df6b7368a54d0af7d2cdd45c0c9bad0999d58c144cb99927a3f76683652b00f2 docker.io/prom/alertmanager:v0.16.2 "/bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --cluster.peer=fl31ca104ja0201.deacluster.intel.com:9094 --config.file=/etc/alertmanager/alertmanager.yml" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-alertmanager-fl31ca104ja0201 aa2055733fe8d426312af5572c94558e89e7cf350e7baba2c22eb6a0e20682fc docker.io/prom/prometheus:v2.7.2 "/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.listen-address=:9095 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://fl31ca104ja0201:9095"; 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-prometheus-fl31ca104ja0201 a9526f50dfacad47af298c0c1b2cf6cfd74b796b6df1945325529c79658d7356 docker.io/prom/node-exporter:v0.17.0 "/bin/node_exporter --no-collector.timex --web.listen-address=:9100 --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs" 10 days ago Up 10 days ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-node-exporter-fl31ca104ja0201 440926ce479bdd114f43e3228cc8cbfe48b4e1a6c2c7fab58c4cd103bc0f3a0e docker.io/arcts/keepalived "./init.sh" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-rgw-default-default-fl31ca104ja0201-yiasjs 2813ca859a7ba0de7fcb6be74a00b9b11a23e79636c5f35fb2b6b4be31a29f89 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-rgw-default-default-fl31ca104ja0201-yvwsmz d68e2f68c45f2ea9a10267c8d964c2aaf026b4291918f4f3fb306da20a532db9 docker.io/arcts/keepalived "./init.sh" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-nfs-nfs-1-fl31ca104ja0201-dsynjg 40f3c0b7455f5540fdb4f428bef4e9032b0ff0f50d302352551abb208eff1f28 docker.io/library/haproxy:2.3 "docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg" 3 weeks ago Up 3 weeks ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-nfs-nfs-1-fl31ca104ja0201-zdbzvv From: Ben <ruidong.gao@xxxxxxxxx> Sent: Wednesday, May 17, 2023 6:32 PM To: Adiga, Anantha <anantha.adiga@xxxxxxxxx> Cc: ceph-users@xxxxxxx Subject: Re: Grafana service fails to start due to bad directory name after Quincy upgrade use this to get relevant long lines in log: journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 | less -S it is '--user 472' by content of unit.run, not the default ceph user 167. Maybe set the directory owner to 472 could help. Hope it helps Ben Adiga, Anantha <anantha.adiga@xxxxxxxxx<mailto:anantha.adiga@xxxxxxxxx>> 于2023年5月18日周四 01:15写道： Ben, Thanks for the suggestion. Changed the user and group to 167 for all files in the data and etc folders in the grafana service folder were not 167. Did a systemctl daemon-reload and restarted the grafana service , but still seeing the same error -- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02 UTC. -- May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:44 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled restart job, restart > May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process exited, code=exite> May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No > May 17 17:07:54 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with result 'exit-code'. ESCOC 2 UTC. -- a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Scheduled restart job, restart counter is at 3. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Main process exited, code=exited, status=127/n/a b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>: Failed with result 'exit-code'. ~ Thank you, Anantha From: Ben <ruidong.gao@xxxxxxxxx<mailto:ruidong.gao@xxxxxxxxx>> Sent: Wednesday, May 17, 2023 2:29 AM To: Adiga, Anantha <anantha.adiga@xxxxxxxxx<mailto:anantha.adiga@xxxxxxxxx>> Cc: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> Subject: Re: Grafana service fails to start due to bad directory name after Quincy upgrade you could check owner of /var/lib/ceph on host with grafana container running. If its owner is root, change to 167:167 recursively. Then systemctl daemon-reload and restart the service. Good luck. Ben Adiga, Anantha <anantha.adiga@xxxxxxxxx<mailto:anantha.adiga@xxxxxxxxx>> 于2023年5月17日周三 03:57写道： Hi Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th However, Grafana fails to start due to bad folder path :/tmp# journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25 -- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51 UTC. -- May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:00 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:10 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main process ex> May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash: /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u> May 16 19:05:20 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Scheduled resta> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Start request r> May 16 19:05:30 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Failed with res> May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ESCOC 19:07:51 UTC. -- 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 3. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Main process exited, code=exited, status=127/n/a ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such file or directory -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Failed with result 'exit-code'. -be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>: Scheduled restart job, restart counter is at 4. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. 31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e. Check if path exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run ls: cannot access '/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run': No such file or directory Check if grafana.fl31ca104ja0201 directory exists: #<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#> ls -l rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run total 36 drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc -rw------- 1 167 167 48 Apr 20 08:13 unit.created -rw------- 1 167 167 390 May 8 16:12 unit.stop -rw------- 1 167 167 390 May 8 16:12 unit.poststop -rw------- 1 167 167 365 May 8 16:12 unit.meta -rw------- 1 167 167 32 May 8 16:12 unit.image -rw------- 1 167 167 38 May 8 16:12 unit.configured -rw------- 1 167 167 1063 May 8 16:12 unit.run cat unit.run set -e # grafana.fl31ca104ja0201 ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null ! /usr/bin/docker rm -f ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init --name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user 472 -e CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> -e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z -v /var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4> Thank you, Anantha _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |