On 06/12/2018 17:10, David Woodhouse wrote: > On Wed, 2018-11-28 at 08:44 -0800, Andy Lutomirski wrote: >>> Can we assume it's always from kernel? The Xen code definitely seems to >>> handle invoking this from both kernel and userspace contexts. >> I learned that my comment here was wrong shortly after the patch landed :( > Turns out the only place I see it getting called from is under > __context_switch(). > > #7 [ffff8801144a7cf0] new_xen_failsafe_callback at ffffffffa028028a [kmod_ebxfix] > #8 [ffff8801144a7d90] xen_hypercall_update_descriptor at ffffffff8100114a > #9 [ffff8801144a7db8] xen_hypercall_update_descriptor at ffffffff8100114a > #10 [ffff8801144a7df0] xen_mc_flush at ffffffff81006ab9 > #11 [ffff8801144a7e30] xen_end_context_switch at ffffffff81004e12 > #12 [ffff8801144a7e48] __switch_to at ffffffff81016582 > #13 [ffff8801144a7ea0] __schedule at ffffffff815d2b37 > > That …114a in xen_hypercall_update_descriptor is the 'pop' instruction > right after the syscall; it's happening when Xen is preempting the > domain in the hypercall and then reloads the segment registers to run > that vCPU again later. > > [ 44185.225289] WARN: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000abbd76060 > > The update_descriptor hypercall args (rdi, rsi) were 0xabbd76060 and 0 > respectively — it was setting a descriptor at that address, to zero. > > Xen then failed to load the selector 0x63 into the %gs register (since > that descriptor has just been wiped?), leaving it zero. > > [ 44185.225256] WARN: xen_failsafe_callback from xen_hypercall_update_descriptor+0xa/0x40 > [ 44185.225263] WARN: DS: 2b/2b ES: 2b/2b FS: 0/0 GS:0/63 > > This is on context switch from a 32-bit task to idle. So > xen_failsafe_callback is returning to the "faulting" instruction, with > a comment saying "Retry the IRET", but in fact is just continuing on > its merry way with %gs unexpectedly set to zero. > > In fact I think this is probably fine in practice, since it's about to > get explicitly set a few lines further down in __context_switch(). But > it's odd enough, and far enough away from what's actually said by the > comments, that I'm utterly unsure. > > In xen_load_tls() we explicitly only do the lazy_load_gs(0) for the > 32-bit kernel. Is that really right? Basically - what is happening is that xen_load_tls() is invalidating the %gs selector while %gs is still non-NUL. If this happens to intersect with a vcpu reschedule, %gs (being non-NUL) takes precedence over KERNGSBASE, and faults when Xen tries to reload it. This results in the failsafe callback being invoked. I think the correct course of action is to use xen_load_gs_index(0) (poorly named - it is a hypercall which does swapgs; mov to %gs; swapgs) before using update_descriptor() to invalidate the segment. That will reset %gs to 0 without touching KERNGSBASE, and can be queued in the same multicall as the update_descriptor() hypercall. ~Andrew
