Host is Ubuntu 20.04 - 5.4.0-26-generic
KVM is 1:4.2-3ubuntu6.17
Guest OS -- various flavors of Windows 2012 server or greater
Symptom: Guests will appear to be running fine for some time (hours
or days), occasionally a perfectly running guest will "lock up" and
become unresponsive to network, VNC, etc.. We are unable to access
the VM and the only resort is to destroy the VM - shutdown will not
tear down the VM.
Looking at a number of things, we can see the following via trace_pipe
-- nothing else is emitted while in the hung state.
cat /sys/kernel/debug/tracing/trace_pipe | grep -i <pid>
.....
<...>-3441924 [004] .... 1907374.582975: kvm_set_irq: gsi 0 level 1 source 0
<...>-3441924 [004] .... 1907374.582977: kvm_pic_set_irq:
chip 0 pin 0 (edge|masked)
<...>-3441924 [004] .... 1907374.582979:
kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge)
<...>-3441924 [004] .... 1907374.582981:
kvm_ioapic_set_irq: pin 2 dst 1 vec 209 (Fixed|logical|edge)
<...>-3441924 [004] .... 1907374.582983: kvm_set_irq: gsi 0
level 0 source 0
<...>-3441924 [004] .... 1907374.582984: kvm_pic_set_irq:
chip 0 pin 0 (edge|masked)
<...>-3441924 [004] .... 1907374.582984:
kvm_ioapic_set_irq: pin 2 dst 1 vec 209 (Fixed|logical|edge)
<...>-3441924 [004] .... 1907374.583457: kvm_set_irq: gsi 0
level 1 source 0
<...>-3441924 [004] .... 1907374.583459: kvm_pic_set_irq:
chip 0 pin 0 (edge|masked)
<...>-3441924 [004] .... 1907374.583461:
kvm_apic_accept_irq: apicid 0 vec 209 (Fixed|edge)
<...>-3441924 [004] .... 1907374.583462:
kvm_ioapic_set_irq: pin 2 dst 1 vec 209 (Fixed|logical|edge)
<...>-3441924 [004] .... 1907374.583464: kvm_set_irq: gsi 0
level 0 source 0
.....
----------------------
Strace (15 second) output:
55.39 0.082181 16 4937 ioctl
23.05 0.034208 14 2390 ppoll
21.56 0.031990 13 2379 futex
0.00 0.000001 0 30 read
------ ----------- ----------- --------- --------- ----------------
100.00 0.148380 9736 total
-----------------------
KVM stat live output:
APIC_ACCESS 11491 44.50% 5.12% 1.82us
5021.41us 9.41us ( +- 9.87% )
EPT_MISCONFIG 8007 31.01% 17.51% 11.25us
11003.56us 46.18us ( +- 8.59% )
HLT 3804 14.73% 76.26% 2.19us
3962.04us 423.30us ( +- 0.44% )
EXTERNAL_INTERRUPT 2492 9.65% 0.46% 0.96us
163.72us 3.91us ( +- 4.36% )
PAUSE_INSTRUCTION 14 0.05% 0.00% 1.24us
6.59us 3.40us ( +- 14.61% )
IO_INSTRUCTION 8 0.03% 0.64% 8.34us
13470.05us 1693.58us ( +- 99.34% )
PENDING_INTERRUPT 3 0.01% 0.00% 2.31us
3.69us 2.99us ( +- 13.36% )
EPT_VIOLATION 2 0.01% 0.00% 7.74us
9.37us 8.56us ( +- 9.51% )
EXCEPTION_NMI 1 0.00% 0.00% 3.09us
3.09us 3.09us ( +- 0.00% )
TPR_BELOW_THRESHOLD 1 0.00% 0.00% 2.39us
2.39us 2.39us ( +- 0.00% )
We have tried quite a few of the HyperV enlightenments and clock
settings and those do not resolve the issue
We can reproduce the issue by stopping all guests(7) on the host and
starting them up at the same time - this may take a couple of
cycles(2-3) en-masse restarts and 1 or 2 out of the 7 will exhibit
this behavior
CPU is host-passthrough
Server has 24 cores with 256G of ram
Looking to see if anyone in the community has encountered this problem.
Thanks
Bill
