On 7/20/21 11:17 AM, Peter Krempa wrote: > On Tue, Jul 20, 2021 at 15:52:58 +0800, Jiatong Shen wrote: >> Hello community, >> >> I am seeing following log in production, >> >> 2021-07-20 07:43:49.417+0000: 3918294: error : >> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot >> acquire state change lock (held by qemuProcessReconnect) >> 2021-07-20 07:44:19.424+0000: 3918296: warning : >> qemuDomainObjBeginJobInternal:4933 : Cannot start job (modify, none) for >> domain instance-0000074e; current job is (modify, none) owned by (3919429 >> qemuProcessReconnect, 0 <null>) for (2183193s, 0s) >> 2021-07-20 07:44:19.424+0000: 3918296: error : >> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot >> acquire state change lock (held by qemuProcessReconnect) >> 2021-07-20 07:44:49.428+0000: 3918296: warning : >> qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for >> domain instance-0000074e; current job is (modify, none) owned by (3919429 >> qemuProcessReconnect, 0 <null>) for (2183223s, 0s) >> 2021-07-20 07:44:49.428+0000: 3918296: error : >> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot >> acquire state change lock (held by qemuProcessReconnect) >> 2021-07-20 07:45:19.429+0000: 3918298: warning : >> qemuDomainObjBeginJobInternal:4933 : >> >> I am confused about what is qemuProcessReconnect and why it acquires a >> domain state lock.. > > qemuProcessReconnect is an operation that is executed in a separate > thread which re-establishes connection to a qemu process once you > restart libvirtd. > > This usually means that the reconnection process got stuck for some > reason. Unfortunately your log doesn't show why and unless you've got a > debug log prior to that happening it won't be possible. > > Theoretically seeing which function the thread doing the reconnect is > stuck at could perhaps show why. This should be pretty easy to see though. All that's needed is attaching gdb an running backtrace. Ideally over all threads as you don't know which one is stuck trying to reconnect. Something like the following: gdb -q -p $(pgrep libvirtd) -ex 't a a bt' -ex 'q' Just make sure you have debuginfo installed for libvirt otherwise the stacktrace won't be useful. Michal
