On Fri, 2021-12-17 at 11:48 -0600, Tom Lendacky wrote:
> On 12/16/21 6:13 PM, David Woodhouse wrote:
> > On Thu, 2021-12-16 at 16:52 -0600, Tom Lendacky wrote:
> > > On baremetal, I haven't seen an issue. This only seems to have a problem
> > > with Qemu/KVM.
> > >
> > > With 191f08997577 I could boot without issues with and without the
> > > no_parallel_bringup. Only after I applied e78fa57dd642 did the failure happen.
> > >
> > > With e78fa57dd642 I could boot 64 vCPUs pretty consistently, but when I
> > > jumped to 128 vCPUs it failed again. When I moved the series to
> > > df9726cb7178, then 64 vCPUs also failed pretty consistently.
> > >
> > > Strange thing is it is random. Sometimes (rarely) it works on the first
> > > boot and then sometimes it doesn't, at which point it will reset and
> > > reboot 3 or 4 times and then make it past the failure and fully boot.
> >
> > Hm, some of that is just artifacts of timing, I'm sure. But now I'm
> > staring at the way that early_setup_idt() can run in parallel on all
> > CPUs, rewriting bringup_idt_descr and loading it.
> >
> > To start with, let's try unlocking the trampoline_lock much later,
> > after cpu_init_exception_handling() has loaded the real IDT.
> >
> > I think we can probably make secondaries load the real IDT early and
> > never use bringup_idt_descr at all, can't we? But let's see if this
> > makes it go away, to start with...
> >
>
> This still fails. I ran with -d cpu_reset on the command line and will
> forward the full log to you. I ran "grep "[ER]IP=" stderr.log | uniq -c"
> and got:
>
> 128 EIP=00000000 EFL=00000000 [-------] CPL=0 II=0 A20=0 SMM=0 HLT=0
> 128 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are before running any of the vCPUs.
>
> 1 RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This is where vCPU0 is at the time of the reset. This address tends to
> be different all the time and so I think it is just where it happens to
> be when the reset occurs and isn't contributing to the reset.
I note that one is in native_write_msr() though. I wonder what it's writing?
Do you have console output (perhaps with earlyprintk=ttyS0) to go with this?
> 5 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> 1 RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> 15 RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> These are some of the APs and all are in wait_for_master_cpu().
As is right and proper. They should be coming up to that point and
waiting for the... erm... controlling CPU to tell them to go any
further.
> 1 EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This seems ok because: CS =9900 00099000 0000ffff 00009b00
> So likely in the trampoline code.
Yeah, that'll be in the bitlock waiting for its turn through the real
mode stack.
1010: 66 0f ba 26 18 btw $0x18,(%esi)
1015: 40 inc %eax
1016: 00 73 04 add %dh,0x4(%ebx)
1019: f3 90 pause
101b: eb f3 jmp 1010 <trampoline_start+0x10>
> 1 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> This one seems odd... could it be the one causing the reset?
> CS =f000 ffff0000 0000ffff 00009a00
Yeah. I'm finding it slightly easier without the 'uniq'...
> CPU Reset (CPU 0)
> RIP=ffffffff810705c6 RFL=00000206 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 1)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 2)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 3)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 4)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 5)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 6)
> RIP=ffffffff8104af06 RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 7)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 8)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 9)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 10)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 11)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 12)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 13)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 14)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 15)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 16)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 17)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 18)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 19)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 20)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 21)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
All those came up and are waiting in wait_for_master_cpu() as they
should.
> CPU Reset (CPU 22)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
That one's in the bitlock, also waiting.
> CPU Reset (CPU 23)
> EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
This one we suspect. Is this what a triple-fault would look like? Not
if it's *already* at f000:fff0, surely?
CPU Reset (CPU 23)
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00800f12
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009a00
SS =0000 00000000 0000ffff 00009200
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008300
GDT= 00000000 0000ffff
IDT= 00000000 0000ffff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000000 CCO=DYNAMIC
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000
XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000
XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000
XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000
> CPU Reset (CPU 24)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 25)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 26)
> RIP=ffffffff8104aefb RFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
These ones made it through the real mode first and are also waiting.
> CPU Reset (CPU 27)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 28)
> EIP=0000101b EFL=00000003 [------C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> CPU Reset (CPU 29)
Still in the real mode bitlock. And after this point they are still
halted in presumably 32-bit BIOS code because the BSP hasn't even
touched them yet.
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 30)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 31)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
> CPU Reset (CPU 32)
> ...
> CPU Reset (CPU 127)
> EIP=3f36e11b EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=1
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
