Just a brief update:
As soon as I umount cephfs virsh is able to talk to libvirtd.
I tested the cephfs with:
df (no problem)
dd if=/dev/zero of=/cephstorage/a.img bs=1G count=1 oflag=dsync (no
problem created random1G file, no I/O issues.)
After mounting cephfs and restarting libvirtd "virsh" hangs again.
Obviously virsh and libvirtd don't like the cephfs mount.
I am just starting to debug the potential problem with cephfs and
libvirtd/virsh.
I originally noted when this problem occurred on two hypervisors the
problem occurred a couple days a part which matched some updates that took
place. I have not tried rolling back patches yet.
I am curious if anyone uses cephfs filesystem and had similar problems
recently.
I will update the form if I find a solution.
Thanks
jerry
Jerry Buburuz
>
> Just found my issue.
>
> After I removed the cephfs mounts it worked!
>
> I will debug ceph.
>
> I assumed because I could touch files on mounted cephfs it was working.
>
> Now virsh list works!
>
> thanks
> jerry
>
> Lars Kellogg-Stedman
>> On Tue, Jun 06, 2023 at 04:56:38PM -0400, Jerry Buburuz wrote:
>>> Recently both virsh stopped talking to the libvirtd. Both stopped
>>> within
>>> a
>>> few days of each other.
>>
>> I've run into exactly the same problem.
>>
>> I'm running libvirt (libvirt-9.0.0-3.fc38.x86_64) on Fedora 38. On
>> Fedora, libvirtd is configured by default to use socket activation and
>> is run with the `--timeout 120` option.
>>
>> After some recent upgrades, I'm seeing the exact same symptoms that
>> Jerry described -- virsh commands simply get stuck at same call to
>> `poll()`.
>>
>> It looks like libvirtd is either crashing or failing to start, because
>> when virsh is in this state the `libvirtd` process isn't running. This
>> makes it *sound* like a systemd problem, but I'm not seeing errors
>> anywhere -- either from libvirtd or from systemd.
>>
>> I've worked around the problem locally by re-configuring libvirtd to
>> run persistently rather than using socket activation:
>>
>> systemctl disable --now libvirtd{,-ro,-admin}.socket
>>
>> cat > /etc/systemd/system/libvirtd.service.d/override.conf <<EOF
>> [Service]
>> EnvironmentFile=
>> EOF
>>
>> systemctl restart libvirtd
>>
>> Package versions in case this helps correlate something:
>>
>> - libvirt-9.0.0-3.fc38.x86_64
>> - systemd-253.5-1.fc38.x86_64
>> - kernel-6.3.6-200.fc38.x86_64
>>
>> Libvirt uri: qemu:///system
>>
>> --
>> Lars Kellogg-Stedman <lars@xxxxxxxxxx> | larsks @ {irc,twitter,github}
>> http://blog.oddbit.com/ | N1LKS
>>
>>
>
>
