On Wed, Jun 11, 2014 at 8:42 PM, Zhang Haoyu <zhanghy@xxxxxxxxxxx> wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> According to this: >>>>>>>>>>> >>>>>>>>>>> https://github.com/torvalds/linux/commit/0d1de2d901f4ba0972a3886496a44fb1d3300dbd >>>>>>>>>>> >>>>>>>>>>> It looks like kvm have been supporting x2apic since kernel 2.6.32, or >>>>>>>>>>> even earlier. >>>>>>>>>>> >>>>>>>>>> This patch is to emulate x2apic for guest, then guest can benefit from some advantages of x2apic, like using RDMSR/WRMSR instead of mmio. >>>>>>>>>> >>>>>>>>>>> However, this following patch: >>>>>>>>>>> >>>>>>>>>>> https://github.com/torvalds/linux/commit/8d14695f9542e9e0195d6e41ddaa52c32322adf5 >>>>>>>>>>> >>>>>>>>>>> Also claims that adding support for x2apic. But this later patch was >>>>>>>>>>> for kernel 3.9. >>>>>>>>>> This patch is to support virtual x2apic mode, which can virtualizing MSR-based APIC accesses by configuring MSR bitmaps, >>>>>>>>>> you can specify some MSR-based accesses without VM exit, the other MSR-based accesses with VM exit, >>>>>>>>>> which belongs to APIC virtualization from certain angle. >>>>>>>>> Thanks Haoyu, I am reading the Intel x2apic manual. >>>>>>>>> >>>>>>>>> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >>>>>>>>> >>>>>>>>> but I don't see the so called "virtual x2apic mode", it seems that the >>>>>>>>> manual does not mention anything about that. Do you mean that when the >>>>>>>>> Guest OS sets bit 10 (x2apic mode enable bit) of the it virtualized >>>>>>>>> IA32_APIC_BASE MSR, it is entering a virtual x2apic mode? But isn't >>>>>>>>> this the same as the aforementioned "emulate x2apic for guest"? Or, >>>>>>>>> "emulate x2apic for guest" is the foundation of "support virtual >>>>>>>>> x2apic mode"? >>>>>>>>> >>>>>>>>> -Jidong >>>>>>>>> >>>>>>>>Oh, I found the answer, the virtualized x2apic mode is not defined in >>>>>>>>the x2apic manual, but is defined in the intel SDM manual, chapter 29 >>>>>>>>- APIC virtualization and virtual interrupts. Ignore my questions >>>>>>>>please, but thank you anyway. >>>>>>>> >>>>>>> Yes, virtual x2apic mode is part of APICv. >>>>>>> Guest has no idea about it is running in a virtual machine, >>>>>>> VMM also prevent guest from enabling/disabling virtual x2apic mode or some other virtualization configurations. >>>>>>> >>>>>> >>>>>>Regarding this "VMM also prevent guest from enabling/disabling virtual >>>>>>x2apic mode", do you mean that guest is not allowed to do something >>>>>>like: >>>>>> >>>>>>Assuming we are in the guest, and we see the original value of the >>>>>>IA32_APICBASE MSR is 0xfee00900, which means EN bit and BSP bit was >>>>>>enabled. And now, we try to set bit 10 (EXTD bit): >>>>>> >>>>>>wrmsr 0x1b 0xfee00d00. >>>>>> >>>>>>Do you mean this wrmsr operation is not allowed? If fact, I tried >>>>>>this, and this simply crashes the guest kernel. (I am using Linux >>>>>>2.6.34 as the guest kernel, and Linux 3.14 as the host kernel.) And >>>>>>this is the exact reason that I initialized this whole discussion. My >>>>>>feeling is this might be a kvm bug, so I wish to figure out and try to >>>>>>fix it. >>>>>> >>>>> The MSR address range 800H through BFFH is architecturally reserved and dedicated for accessing APIC registers in x2APIC mode, >>>>> you are writing the wrong MSR address. >>>>> Please see SDN 3A 10.12.1.2 x2APIC Register Address Space, you will get the answer. >>>>> >>>>Oh, I was writing to 0x1b, which is the address of IA32_APIC_BASE MSR. >>>>In particular I was trying to write bit 10, which is the x2APIC >>>>enable/disable bit. As I said, I just wish to enable that x2APIC. The >>>>original value 0xfee00900 suggests x2APIC is not enabled, i.e., bit 10 >>>>is 0. According to SDM manual, this bit has to be 1 if we want to >>>>enable x2APIC. >>>> >>> Sorry, I misread it. >>> Please try >>> u64 val; >>> rdmsrl(0x0000001b, val); >>> if ( !(val & (1UL << 10)) ) >>> wrmsrl(0x0000001b, val | (1UL << 10)); >>> >>> If the crash still happened, maybe other settings is needed, like switching apic to physical x2apic, I'm not sure. >>> Please report the crash message. >>> >>Yes I wrote a simple kernel module which just includes the above four >>lines in the module init function, and when I try to install the >>module with insmod command, the crash still happens, the crash message >>is: >> >>[ 1025.424260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >>task.ti=f58b2000) >>[ 1025.425093] Stack: >>[ 1025.425093] Call Trace: >>[ 1025.425093] Leftover inexact backtrace: >>[ 1025.425093] >>[ 1025.425093] Code: 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba 94 c0 >>8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba 80 00 >>00 00 89 f0 <e8> 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 8d bf >>00 00 >> > Does the vcpu have x2apic feature? > please make a confirmation by 'cat /proc/cpuinfo | grep -i x2apic' > Configure crashkernel to kdump more info, please. > > Thanks, > Zhang Haoyu Hi, Haoyu, I think I have found the root cause. As I said I was using 2.6.34 kernel in the guest, and CONFIG_X86_X2APIC was not enabled by default in my kernel. After I set CONFIG_X86_X2APIC and recompile the kernel, the new kernel just works fine in x2apic mode. Thanks for your time and help. -Jidong >>And this above message will be displayed repeatedly all the time and >>the system will become unresponsive. >>[ 1090.922260] Process bash (pid: 3091, ti=f58b2000 task=f5a725f0 >>task.ti=f58b2000) >>[ 1090.923089] Stack: >>[ 1090.923089] Call Trace: >>[ 1090.923089] Leftover inexact backtrace: >>[ 1090.923089] >>[ 1090.923089] Code: 83 c4 10 5b 5e 5f 5d e9 98 4a 42 00 8b 0d 3c ba >>94 c0 8d 93 f0 00 00 00 89 f0 ff 51 78 eb 07 90 8d 74 26 00 f3 90 ba >>80 00 00 00 <89> f0 e8 12 01 1d 00 85 c0 74 ee eb ad 8d b6 00 00 00 00 >>8d bf >> >> >> >>>>>>>>>>> >>>>>>>>>>> So I am very confused: >>>>>>>>>>> >>>>>>>>>>> First, what's the difference between these two patches? Or say, does >>>>>>>>>>> kvm support x2apic since kernel 2.6.32, or since kernel 3.9? >>>>>>>>>>> >>>>>>>>>> kvm support x2apic since kernel 2.6.32, but not support virtual x2apic mode until kernel 3.9. >>>>>>>>>> >>>>>>>>>>> Second, can guest use x2apic even if the host does not? (Assuming qemu >>>>>>>>>>>has exposed this feature to guest.) The word "use" means something >>>>>>>>>>> like, accessing x2apic registers, setting the x2apic enable bit in >>>>>>>>>>> IA32_APICBASE MSR (i.e. bit 10). >>>>>>>>>>> >>>>>>>>>> Yes, you can use x2apic even if the PCPU does not support, and benefit from it, like performance bonus from MSR accesses instead of MMIO. >>>>>>>>>> But, if I remember correctly, if your guest does not support interrupt-remmping, you only can use physical destination mode when using x2apic, >>>>>>>>>> please see enable_IR_x2apic(). >>>>>>>>>> >>>>>>>>>>> -Jidong > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
