Hello Users, We're using libvirt with KVM and the orchestrator is Cloudstack. I raised the issue already at Cloudstack at https://github.com/apache/cloudstack/issues/8211 but appears to be at libvirtd. Did the same in libvirt ML at https://lists.libvirt.org/archives/list/users@xxxxxxxxxxxxxxxxx/thread/SA2I4QZGVVEIKPJU7E2KAFYYFZLJZDMV/ but I'm now here looking for answers. Below is our environment & issue description: Ceph: v17.2.0 Pool: replicated Number of block images in this pool: more than 1250 # virsh pool-info c15508c7-5c2c-317f-aa2e-29f307771415 Name: c15508c7-5c2c-317f-aa2e-29f307771415 UUID: c15508c7-5c2c-317f-aa2e-29f307771415 State: running Persistent: no Autostart: no Capacity: 1.25 PiB Allocation: 489.52 TiB Available: 787.36 TiB # kvm --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.27) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers # libvirtd --version libvirtd (libvirt) 6.0.0 It appears that one of our Cloudstack KVM clusters having 8 hosts is having the issue. We have HCI on these 8 hosts and there are around 700+ VMs running. But strange enough, there are these logs like below on hosts. Oct 25 13:38:11 hv-01 libvirtd[9464]: failed to open the RBD image '087bb114-448a-41d2-9f5d-6865b62eed15': No such file or directory Oct 25 20:35:22 hv-01 libvirtd[9464]: failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory Oct 26 09:48:33 hv-01 libvirtd[9464]: failed to open the RBD image 'a3fe82f8-afc9-4604-b55e-91b676514a18': No such file or directory We've got DNS servers on which there is an`A` record resolving to all the IPv4 Addresses of 5 monitors and there have not been any issues with the DNS resolution. But the issue of "failed to open the RBD image 'ccc1168a-5ffa-4b6d-a953-8e0ac788ebc5': No such file or directory" gets more weird because the VM that is making use of that RBD image lets say "087bb114-448a-41d2-9f5d-6865b62eed15" is running on an altogether different host like "hv-06". On further inspection of that specific Virtual Machine, it has been running on that host "hv-06" for more than 4 months or so. Fortunately, the Virtual Machine has no issues and has been running since then. There are absolutely no issues with any of the Virtual Machines because of these warnings. >From libvirtd mailing lists, one of the community members helped me understand that libvirt only tries to get the info of the images and doesn't open for reading or writing. All hosts where there is libvirtd tries doing the same. We manually did "virsh pool-refresh" which CloudStack itself takes care of at regular intervals and the warning messages still appear. Please help me find the cause and let me know if further information is needed. Thanks, Jayanth Reddy _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |