Hi,
after converting my 2 node cluster to cephadm I'm in lots of trouble.
- containerized mds are not available in the cluster. I must
run mds from systemd to make my fs available.
- osd podman objects are not found after a reboot of one node. I
don't want to test it on the second node, because if it looses
the podman config as well, my cluster is dead!
Can anybody help me?
Best regards Magnus
Output from cephadm.log:
cephadm ['--image', 'docker.io/ceph/ceph:v15', '--no-container-init', 'ls']
2021-10-25 22:47:01,929 DEBUG container_init=False
2021-10-25 22:47:01,929 DEBUG Running command: systemctl is-enabled
ceph-mds@s1
2021-10-25 22:47:01,935 DEBUG systemctl: stdout enabled
2021-10-25 22:47:01,935 DEBUG Running command: systemctl is-active
ceph-mds@s1
2021-10-25 22:47:01,940 DEBUG systemctl: stdout active
2021-10-25 22:47:01,940 DEBUG Running command: ceph -v
2021-10-25 22:47:02,009 DEBUG ceph: stdout ceph version 15.2.13
(c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
2021-10-25 22:47:02,016 DEBUG Running command: systemctl is-enabled
ceph-osd@1
2021-10-25 22:47:02,024 DEBUG systemctl: stdout disabled
2021-10-25 22:47:02,024 DEBUG Running command: systemctl is-active
ceph-osd@1
2021-10-25 22:47:02,031 DEBUG systemctl: stdout inactive
2021-10-25 22:47:02,031 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
2021-10-25 22:47:02,036 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,036 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
2021-10-25 22:47:02,041 DEBUG systemctl: stdout active
2021-10-25 22:47:02,041 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,068 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,069 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mon.s1
2021-10-25 22:47:02,111 DEBUG /bin/podman: stdout
1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
22:45:39.315992691 +0200 CEST,
2021-10-25 22:47:02,203 DEBUG Running command: /bin/podman exec
1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48 ceph -v
2021-10-25 22:47:02,356 DEBUG /bin/podman: stdout ceph version 15.2.13
(c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
2021-10-25 22:47:02,434 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
2021-10-25 22:47:02,440 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,440 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
2021-10-25 22:47:02,445 DEBUG systemctl: stdout active
2021-10-25 22:47:02,445 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,468 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,470 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mgr.s1
2021-10-25 22:47:02,513 DEBUG /bin/podman: stdout
2b08ddb6182e14985939e50a00e2306a77c3068a65ed122dab8bd5604c91af65,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
22:45:39.535973449 +0200 CEST,
2021-10-25 22:47:02,601 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
2021-10-25 22:47:02,607 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,607 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
2021-10-25 22:47:02,612 DEBUG systemctl: stdout activating
2021-10-25 22:47:02,612 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,636 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,637 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0
2021-10-25 22:47:02,709 DEBUG /bin/podman: stderr Error: error
inspecting object: no such object:
"ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0"
2021-10-25 22:47:02,712 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.2
2021-10-25 22:47:02,718 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,718 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.2
2021-10-25 22:47:02,723 DEBUG systemctl: stdout activating
2021-10-25 22:47:02,723 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,747 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,748 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2
2021-10-25 22:47:02,821 DEBUG /bin/podman: stderr Error: error
inspecting object: no such object:
"ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2"
2021-10-25 22:47:02,823 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.6
2021-10-25 22:47:02,830 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,830 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.6
2021-10-25 22:47:02,835 DEBUG systemctl: stdout activating
2021-10-25 22:47:02,835 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,860 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,861 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6
2021-10-25 22:47:02,948 DEBUG /bin/podman: stderr Error: error
inspecting object: no such object:
"ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6"
2021-10-25 22:47:02,950 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.7
2021-10-25 22:47:02,956 DEBUG systemctl: stdout enabled
2021-10-25 22:47:02,957 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.7
2021-10-25 22:47:02,961 DEBUG systemctl: stdout activating
2021-10-25 22:47:02,962 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:02,986 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:02,988 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7
2021-10-25 22:47:03,063 DEBUG /bin/podman: stderr Error: error
inspecting object: no such object:
"ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7"
2021-10-25 22:47:03,066 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.8
2021-10-25 22:47:03,072 DEBUG systemctl: stdout enabled
2021-10-25 22:47:03,072 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.8
2021-10-25 22:47:03,077 DEBUG systemctl: stdout activating
2021-10-25 22:47:03,077 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:03,099 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:03,101 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8
2021-10-25 22:47:03,174 DEBUG /bin/podman: stderr Error: error
inspecting object: no such object:
"ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8"
2021-10-25 22:47:03,176 DEBUG Running command: systemctl is-enabled
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@xxxxxx.s1.ajxyaf
2021-10-25 22:47:03,182 DEBUG systemctl: stdout enabled
2021-10-25 22:47:03,182 DEBUG Running command: systemctl is-active
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@xxxxxx.s1.ajxyaf
2021-10-25 22:47:03,187 DEBUG systemctl: stdout active
2021-10-25 22:47:03,187 DEBUG Running command: /bin/podman --version
2021-10-25 22:47:03,210 DEBUG /bin/podman: stdout podman version 3.2.3
2021-10-25 22:47:03,212 DEBUG Running command: /bin/podman inspect
--format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
.Config.Labels "io.ceph.version"}}
ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mds.fs.s1.ajxyaf
2021-10-25 22:47:03,256 DEBUG /bin/podman: stdout
818588b3702cbed9ff38abb07ae39e08f9700b8c8740f40f81641d0129bd8338,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
22:45:39.314981699 +0200 CEST,
--
Dr. Magnus Harlander
Mail: harlan@xxxxxxxxx
Web: www.harlan.de
Stiftung: www.harlander-stiftung.de
Ceterum censeo bitcoin esse delendam!
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
