Dear KVM experts, We have two hosts that got stuck, and the last serial port logs had kvm prints vmx: unexpected exit reason 0x3. last logs of HostA: [23031085.916249] kvm [9737]: vcpu6, guest rIP: 0xffffffffb190d1b5 vmx: unexpected exit reason 0x3 [23031085.916251] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. last logs of HostB: [16755112.797211] kvm [2787303]: vcpu11, guest rIP: 0x70a8f4 vmx: unexpected exit reason 0x3 [16755112.797213] kvm [2787303]: vcpu16, guest rIP: 0x70a9ae vmx: unexpected exit reason 0x3 [16755112.797214] kvm [2787303]: vcpu17, guest rIP: 0x70a9ae vmx: unexpected exit reason 0x3 [16755112.797217] kvm [2787303]: vcpu15, guest rIP: 0x70d707 vmx: unexpected exit reason 0x3 [16755112.797219] kvm [2787303]: vcpu12, guest rIP: 0x701431 vmx: unexpected exit reason 0x3 [16755112.797221] kvm [2787303]: vcpu7, guest rIP: 0x70b005 vmx: unexpected exit reason 0x3 [16755112.797222] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797224] kvm [2787303]: vcpu4, guest rIP: 0x796fa6 vmx: unexpected exit reason 0x3 [16755112.797224] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797229] kvm [3588862]: vcpu3, guest rIP: 0xffffffff816c7a1b vmx: unexpected exit reason 0x3 [16755112.797230] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797231] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797231] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797232] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797233] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797235] kvm [9066]: vcpu5, guest rIP: 0xffffffff8a4a1c0e vmx: unexpected exit reason 0x3 [16755112.797236] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797236] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [16755112.797262] kvm [2813867]: vcpu0, guest rIP: 0xffffffff816c7a1b vmx: unexpected exit reason 0x3 [16755112.797263] set kvm_intel.dump_invalid_vmcs=1 to dump internal KVM state. [18446744004.989880] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 [18446744004.989880] PGD 0 P4D 0 [18446744004.989880] Oops: 0000 [#1] SMP NOPTI [18446744004.989880] CPU: 10 PID: 0 Comm: swapper/10 Kdump: loaded Tainted: G OE --------- -t - 4.18.0-193.14.2.el8_2.x86_64 #1 [18446744004.989880] Hardware name: xxxxx, BIOS xx.xx.xxxx 02/18/2020 [18446744004.989880] RIP: 0010:__list_add_valid+0x0/0x50 [18446744004.989880] Code: ff ff 49 c7 07 00 00 00 00 41 c7 47 08 00 00 00 00 48 89 44 24 28 e9 dc fe ff ff 48 89 6c 24 28 e9 d2 fe ff ff e8 20 08 c8 ff <48> 8b 42 08 49 89 d0 48 39 f0 0f 85 8c 00 00 00 48 8b 10 4c 39 c2 Kernel version is: 4.18.0-193.14.2.el8_2.x86_64 CPU is Intel(R) Xeon(R) Gold 6230N CPU @ 2.30GHz When the hosts were found to be stuck, both had been stuck for several days. We tried triggering a panic collection of vmcore using sysrq+c magic key, but there was no response. Eventually, we had to do a hard reboot by pressing the power button to recover. There is no crashdump generated. Before the two hosts got stuck, they both printed vmx: unexpected exit reason 0x3. Looking at the code, we found exit reason 0x3 is EXIT_REASON_INIT_SIGNAL, means that the current CPU received INIT IPI in non-root mode. But found INIT IPI is only sent during CPU setup. Anyone know why INIT IPI is generated? HostB printed NULL pointer BUG, but the panic process did not proceed further and instead got stuck. The time 18446744004.989880 is incorrect, the uptime of HostB is 193 days. We suspect hostB's exception are also related to the previous vmx unexpected exit. Anyone encountered similar cases before? Are there any solutions and suggestions?
