I've created an issue: https://tracker.ceph.com/issues/57918 What can I do more the get to fix this issue? And the output of the requested commands [cephadm@mdshost2 ~]$ sudo lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv_home vg_sys -wi-ao---- 256.00m lv_opt vg_sys -wi-ao---- 3.00g lv_root vg_sys -wi-ao---- 5.00g lv_swap vg_sys -wi-ao---- 7.56g lv_tmp vg_sys -wi-ao---- 1.00g lv_var vg_sys -wi-ao---- 15.00g lv_var_log vg_sys -wi-ao---- 5.00g lv_var_log_audit vg_sys -wi-ao---- 512.00m [cephadm@mdshost2 ~]$ sudo vgs -a VG #PV #LV #SN Attr VSize VFree vg_sys 1 8 0 wz--n- <49.00g 11.68g [cephadm@mdshost2 ~]$ sudo parted --list Model: VMware Virtual disk (scsi) Disk /dev/sda: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1049kB 1075MB 1074MB primary xfs boot 2 1075MB 53.7GB 52.6GB primary lvm Error: /dev/sdb: unrecognised disk label Model: VMware Virtual disk (scsi) Disk /dev/sdb: 53.7GB Sector size (logical/physical): 512B/512B Partition Table: unknown Disk Flags: ________________________________ From: Guillaume Abrioux <gabrioux@xxxxxxxxxx> Sent: Monday, October 24, 2022 5:50:20 PM To: Sake Paulusma <sake1989@xxxxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: Failed to probe daemons or devices Hello Sake, Could you share the output of vgs / lvs commands? Also, I would suggest you to open a tracker [1] Thanks! [1] https://tracker.ceph.com/projects/ceph-volume<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fprojects%2Fceph-volume&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378380450%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QZh7xliaxqBOtNGgJsfUo2OGZV%2FzOI9cdNotTNT%2Bs%2BU%3D&reserved=0> On Mon, 24 Oct 2022 at 10:51, Sake Paulusma <sake1989@xxxxxxxxxxx<mailto:sake1989@xxxxxxxxxxx>> wrote: Last friday I upgrade the Ceph cluster from 17.2.3 to 17.2.5 with "ceph orch upgrade start --image localcontainerregistry.local.com:5000/ceph/ceph:v17.2.5-20221017<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%3Av17.2.5-20221017&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378380450%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GKhA934ODD%2FQEHdd97vLskLkEhJuekIfUjBqx4wRlGU%3D&reserved=0>". After sometime, an hour?, I've got a health warning: CEPHADM_REFRESH_FAILED: failed to probe daemons or devices. I'm using only Cephfs on the cluster and it's still working correctly. Checking the running services, everything is up and running; mon, osd and mds. But on the hosts running mon and mds services I get errors in the cephadm.log, see the loglines below. I look likes cephadm tries to start a container for checking something? What could be wrong? On mon nodes I got the following: 2022-10-24 10:31:43,880 7f179e5bfb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2022-10-24 10:31:44,333 7fc2d52b6b80 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%40sha256%3A122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378380450%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uNhTCkSwBrcq%2FZNFF6TgKHy5nmC%2FDM5ly3QMIEl8Z8I%3D&reserved=0>', 'ceph-volume', '--fsid', '8909ef90-22ea-11ed-8df1-0050569ee1b1', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2022-10-24 10:31:44,663 7fc2d52b6b80 INFO Inferring config /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/mon.oqsoel24332/config 2022-10-24 10:31:44,663 7fc2d52b6b80 DEBUG Using specified fsid: 8909ef90-22ea-11ed-8df1-0050569ee1b1 2022-10-24 10:31:45,574 7fc2d52b6b80 INFO Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%40sha256%3A122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378380450%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uNhTCkSwBrcq%2FZNFF6TgKHy5nmC%2FDM5ly3QMIEl8Z8I%3D&reserved=0> -e NODE_NAME=monnode2.local.com<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmonnode2.local.com%2F&data=05%7C0 1%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378692872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DhsHCxFBck3TkzM%2FBnKFArbX13pWP0x2fowL9NoC7c0%3D&reserved=0> -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/run/ceph:z -v /var/log/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/log/ceph:z -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /tmp/ceph-tmp31tx1iy2:/etc/ceph/ce ph.conf:z localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%40sha256%3A122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378692872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pVgOytiFU6r13Pn1pr30uTRiYAoJNBv%2F9yQtaeNvXwQ%3D&reserved=0> inventory --format=json-pretty --filter-for-batch 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr Traceback (most recent call last): 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module> 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr self.main(self.argv) 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return f(*a, **kw) 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr instance.main() 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 53, in main 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr with_lsm=self.args.with_lsm)) 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 39, in __init__ 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr all_devices_vgs = lvm.get_all_devices_vgs() 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in get_all_devices_vgs 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in <listcomp> 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 517, in __init__ 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr raise ValueError('VolumeGroup must have a non-empty name') 2022-10-24 10:31:45,575 7fc2d52b6b80 INFO /bin/podman: stderr ValueError: VolumeGroup must have a non-empty name On mds nodes I got the following: 2022-10-24 10:25:18,506 7f613f6fdb80 DEBUG -------------------------------------------------------------------------------- cephadm ['gather-facts'] 2022-10-24 10:25:19,047 7fd9b0d92b80 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%40sha256%3A122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378692872%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pVgOytiFU6r13Pn1pr30uTRiYAoJNBv%2F9yQtaeNvXwQ%3D&reserved=0>', 'ceph-volume', '--fsid', '8909ef90-22ea-11ed-8df1-0050569ee1b1', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2022-10-24 10:25:19,388 7fd9b0d92b80 DEBUG Using specified fsid: 8909ef90-22ea-11ed-8df1-0050569ee1b1 2022-10-24 10:25:20,306 7fd9b0d92b80 INFO Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=localcontainerregistry.local.com:5000/ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2Fceph%2Fceph%40sha256%3A122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0&data=05%7C01%7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378849100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1BYancA6xzUHVEJtP9dX8%2B75BXdYsDE173TgxsXzBO8%3D&reserved=0> -e NODE_NAME=mdsnode1.local.com<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmdsnode1.local.com%2F&data=05%7C01% 7C%7C8cf7f4f7348f4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378849100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sJgN8wZU6s21P2RObYsWsWkBuz8N9cIYNU6y3JNr05s%3D&reserved=0> -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/run/ceph:z -v /var/log/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1:/var/log/ceph:z -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8909ef90-22ea-11ed-8df1-0050569ee1b1/selinux:/sys/fs/selinux:ro -v /:/rootfs localcontainerregistry.local.com:5000<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flocalcontainerregistry.local.com%3A5000%2F&data=05%7C01%7C%7C8cf7f4f7348f 4560917308dab5d77e21%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638022234378849100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g6X8qiHxuAFnVN7dWYUF%2BvzJlxlfJdLTBMewDjaVX5A%3D&reserved=0> /ceph/ceph@sha256:122436e2f1df0c803666c5591db4a9b6c9196a71b4d44c6bd5d18102509dfca0 inventory --format=json-pretty --filter-for-batch 2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr Traceback (most recent call last): 2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module> 2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2022-10-24 10:25:20,306 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr self.main(self.argv) 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return f(*a, **kw) 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr instance.main() 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 53, in main 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr with_lsm=self.args.with_lsm)) 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 39, in __init__ 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr all_devices_vgs = lvm.get_all_devices_vgs() 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in get_all_devices_vgs 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 797, in <listcomp> 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr return [VolumeGroup(**vg) for vg in vgs] 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 517, in __init__ 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr raise ValueError('VolumeGroup must have a non-empty name') 2022-10-24 10:25:20,307 7fd9b0d92b80 INFO /bin/podman: stderr ValueError: VolumeGroup must have a non-empty name _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> -- Guillaume Abrioux Senior Software Engineer _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |