Hi Eric, > On 7 Jun 2023, at 16:40, Eric Auger <eauger@xxxxxxxxxx> wrote: > > Hi Miguel, > > On 6/6/23 19:52, Miguel Luis wrote: >> Hello Eric, Marc, >> >>> On 6 Jun 2023, at 09:33, Eric Auger <eauger@xxxxxxxxxx> wrote: >>> >>> Hi Marc, >>> >>> On 5/17/23 16:12, Marc Zyngier wrote: >>>> On Wed, 17 May 2023 09:59:45 +0100, >>>> Eric Auger <eauger@xxxxxxxxxx> wrote: >>>>> >>>>> Hi Marc, >>>>> Hi Marc, >>>>> On 5/16/23 22:28, Marc Zyngier wrote: >>>>>> On Tue, 16 May 2023 17:53:14 +0100, >>>>>> Eric Auger <eauger@xxxxxxxxxx> wrote: >>>>>>> >>>>>>> Hi Marc, >>>>>>> >>>>>>> On 5/15/23 19:30, Marc Zyngier wrote: >>>>>>>> This is the 4th drop of NV support on arm64 for this year. >>>>>>>> >>>>>>>> For the previous episodes, see [1]. >>>>>>>> >>>>>>>> What's changed: >>>>>>>> >>>>>>>> - New framework to track system register traps that are reinjected in >>>>>>>> guest EL2. It is expected to replace the discrete handling we have >>>>>>>> enjoyed so far, which didn't scale at all. This has already fixed a >>>>>>>> number of bugs that were hidden (a bunch of traps were never >>>>>>>> forwarded...). Still a work in progress, but this is going in the >>>>>>>> right direction. >>>>>>>> >>>>>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than >>>>>>>> the L0 IPA space. This fixes a number of subtle issues, depending on >>>>>>>> how the initial guest was created. >>>>>>>> >>>>>>>> - Consequently, the patch series has gone longer again. Boo. But >>>>>>>> hopefully some of it is easier to review... >>>>>>>> >>>>>>>> [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@xxxxxxxxxx >>>>>>> >>>>>>> I have started testing this and when booting my fedora guest I get >>>>>>> >>>>>>> [ 151.796544] kvm [7617]: Unsupported guest sys_reg access at: >>>>>>> 23f425fd0 [80000209] >>>>>>> [ 151.796544] { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write }, >>>>>>> >>>>>>> as soon as the host has kvm-arm.mode=nested >>>>>>> >>>>>>> This seems to be triggered very early by EDK2 >>>>>>> (ArmPkg/Drivers/TimerDxe/TimerDxe.c). >>>>>>> >>>>>>> If I am not wrong this CNTV_CTL_EL0. Do you have any idea? >>>>>> >>>>>> So here's my current analysis: >>>>>> >>>>>> I assume you are running EDK2 as the L1 guest in a nested >>>>>> configuration. I also assume that you are not running on an Apple >>>>>> CPU. If these assumptions are correct, then EDK2 runs at vEL2, and is >>>>>> in nVHE mode. >>>>>> >>>>>> Finally, I'm going to assume that your implementation has FEAT_ECV and >>>>>> FEAT_NV2, because I can't see how it could fail otherwise. >>>>> all the above is correct. >>>>>> >>>>>> In these precise conditions, KVM sets the CNTHCTL_EL2.EL1TVT bit so >>>>>> that we can trap the EL0 virtual timer and faithfully emulate it (it >>>>>> is otherwise written to memory, which isn't very helpful). >>>>> >>>>> indeed >>>>>> >>>>>> As it turns out, we don't handle these traps. I didn't spot it because >>>>>> my test machines are all Apple boxes that don't have a nVHE mode, so >>>>>> nothing on the nVHE path is getting *ANY* coverage. Hint: having >>>>>> access to such a machine would help (shipping address on request!). >>>>>> Otherwise, I'll eventually kill the nVHE support altogether. >>>>>> >>>>>> I have written the following patch, which compiles, but that I cannot >>>>>> test with my current setup. Could you please give it a go? >>>>> >>>>> with the patch below, my guest boots nicely. You did it great on the 1st >>>>> shot!!! So this fixes my issue. I will continue testing the v10. >>>> >>>> Thanks a lot for reporting the issue and testing my hacks. I'll >>>> eventually fold it into the rest of the series. >>>> >>>> By the way, what are you using as your VMM? I'd really like to >>>> reproduce your setup. >>> Sorry I missed your reply. I am using libvirt + qemu (feat Miguel's RFC) >>> and fedora L1 guest. >>> >> >> Following this subject, I’ve forward ported Alexandru’s KUT patches >> ( and I encourage others to do it also =) ) which expose an EL2 test that > > Do you have a branch available with Alexandru's rebased kut series? Now I do :) https://github.com/mluis/kvm-unit-tests/tree/nv-WIP Thanks, Miguel > > Thanks > > Eric >> does three checks: >> >> - whether VHE is supported and enabled >> - disable VHE >> - re-enable VHE >> >> I’m running qemu with virtualization=on as well to run this test and it is passing although >> problems seem to happen when running with virtualization=off, which I’m still looking into it. >> >> Thanks >> Miguel >> >>> Thanks to your fix, this boots fine. But at the moment it does not >>> reboot and hangs in edk2 I think. Unfortunately this time I have no >>> trace on host :-( While looking at your series I will add some traces. >>> >>> Eric >>>> >>>> Cheers, >>>> >>>> M.
