Last friday I upgrade the Ceph cluster from 17.2.3 to 17.2.5 with "ceph orch upgrade start --image localcontainerregistry.local.com:5000/ceph/ceph:v17.2.5-20221017". After sometime, an hour?, I've got a health warning: CEPHADM_REFRESH_FAILED: failed to probe daemons or devices. I'm using only Cephfs on the cluster and it's still working correctly.
Checking the running services, everything is up and running; mon, osd and mds. But on the hosts running mon and mds services I get errors in the cephadm.log, see the loglines below.
I look likes cephadm tries to start a container for checking something? What could be wrong?
On mon nodes I got the following:
2022-10-24 10:31:43,880 7f179e5bfb80 DEBUG --------------------------------------------------------------------------------
cephadm ['gather-facts']
2022-10-24 10:31:44,333 7fc2d52b6b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0', 'ceph-volume', '--fsid', '8909ef90-22ea-11ed-8df1-0050569ee1b1', '--', 'inventory', '--format=json-pretty', '--filter-for-batch']
2022-10-24 10:31:44,663 7fc2d52b6b80 INFO Inferring config /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/mon.oqsoel24332/config
2022-10-24 10:31:44,663 7fc2d52b6b80 DEBUG Using specified fsid: 8909ef90-22ea-11ed-8df1-0050569ee1b1
2022-10-24 10:31:45,574 7fc2d52b6b80 INFO Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0 -e NODE_NAME=monnode2.local.com -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/run/ceph:z -v /var/log/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/log/ceph:z -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /tmp/ceph-tmp31tx1iy2:/etc/ceph/ce
ph.conf:z localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0 inventory --format=json-pretty --filter-for-batch
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr Traceback (most recent call last):
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr self.main(self.argv)
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return f(*a, **kw)
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args)
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr instance.main()
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 53, in main
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr with_lsm=self.args.with_lsm))
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 39, in __init__
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr all_devices_vgs = lvm.get_all_devices_vgs()
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in get_all_devices_vgs
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs]
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in <listcomp>
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs]
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 517, in __init__
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr raise ValueError('VolumeGroup must have a non-empty name')
2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr ValueError: VolumeGroup must have a non-empty name
On mds nodes I got the following:
2022-10-24 10:25:18,506 7f613f6fdb80 DEBUG --------------------------------------------------------------------------------
cephadm ['gather-facts']
2022-10-24 10:25:19,047 7fd9b0d92b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0', 'ceph-volume', '--fsid', '8909ef90-22ea-11ed-8df1-0050569ee1b1', '--', 'inventory', '--format=json-pretty', '--filter-for-batch']
2022-10-24 10:25:19,388 7fd9b0d92b80 DEBUG Using specified fsid: 8909ef90-22ea-11ed-8df1-0050569ee1b1
2022-10-24 10:25:20,306 7fd9b0d92b80 INFO Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0 -e NODE_NAME=mdsnode1.local.com -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/run/ceph:z -v /var/log/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/log/ceph:z -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/selinux:/sys/fs/selinux:ro -v /:/rootfs localcontainerregistry.local.com:5000
/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0 inventory --format=json-pretty --filter-for-batch
2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr Traceback (most recent call last):
2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr self.main(self.argv)
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return f(*a, **kw)
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args)
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr instance.main()
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 53, in main
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr with_lsm=self.args.with_lsm))
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 39, in __init__
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr all_devices_vgs = lvm.get_all_devices_vgs()
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in get_all_devices_vgs
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs]
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in <listcomp>
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs]
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 517, in __init__
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr raise ValueError('VolumeGroup must have a non-empty name')
2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr ValueError: VolumeGroup must have a non-empty name
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
