Re: [PATCH v3 00/21] KVM: arm64: Rewrite page-table code and fault handling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Will,

On 8/25/20 7:39 PM, Will Deacon wrote:
Hello folks,

This is version three of the KVM page-table rework that I previously posted
here:

   v1: https://lore.kernel.org/r/20200730153406.25136-1-will@xxxxxxxxxx
   v2: https://lore.kernel.org/r/20200818132818.16065-1-will@xxxxxxxxxx

Changes since v2 include:

   * Rebased onto -rc2, which includes the conflicting OOM blocking fixes
   * Dropped the patch trying to "fix" the memcache in kvm_phys_addr_ioremap()


It's really nice work, making the code unified/simplified greatly.
However, it seems it doesn't work well with HugeTLBfs. Please refer
to the following test result and see if you have quick idea, or I
can debug it a bit :)

Note: I think the failing cases (FAIL[1] and FAIL[2]) would be
caused by same issue.

Machine	     Host                     Guest              Result
===============================================================
ThunderX2    VA_BITS:   42           PAGE_SIZE:  4KB     Passed
             PAGE_SIZE: 64KB                    64KB     passed
             THP:       disabled
             HugeTLB:   disabled
---------------------------------------------------------------
ThunderX2    VA_BITS:   42           PAGE_SIZE:  4KB     Passed
             PAGE_SIZE: 64KB                    64KB     passed
             THP:       enabled
             HugeTLB:   disabled
----------------------------------------------------------------
ThunderX2    VA_BITS:   42           PAGE_SIZE:  4KB     Fail[1]
             PAGE_SIZE: 64KB                    64KB     Fail[1]
             THP:       disabled
             HugeTLB:   enabled
---------------------------------------------------------------
ThunderX2    VA_BITS:   39           PAGE_SIZE:  4KB     Passed
             PAGE_SIZE: 4KB                     64KB     Passed
             THP:       disabled
             HugeTLB:   disabled
---------------------------------------------------------------
ThunderX2    VA_BITS:   39           PAGE_SIZE:  4KB     Passed
             PAGE_SIZE: 4KB                     64KB     Passed
             THP:       enabled
             HugeTLB:   disabled
--------------------------------------------------------------
ThunderX2    VA_BITS:   39           PAGE_SIZE: 4KB     Fail[2]
             PAGE_SIZE: 4KB                    64KB     Fail[2]
             THP:       disabled
             HugeTLB:   enabled

NOTE: The commands used to start VM are same for FAIL[1] and
FAIL[2] and the host kernel log are similar. So I don't provide
the kernel log for FAIL[2]. I guess they're caused by same
issue.

Fail[1]
===============================================================

start_vm_aarch64_hugetlbfs() {
   echo 16 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages

   /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                     \
   --enable-kvm -machine virt,gic-version=host                                 \
   -cpu host -smp 8,sockets=8,cores=1,threads=1                                \
   -m 4G -mem-prealloc -mem-path /dev/hugepages                                \
   -monitor none -serial mon:stdio -nographic -s                               \
   -bios /home/gavin/sandbox/qemu.main/pc-bios/edk2-aarch64-code.fd            \
   -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image               \
   -initrd /home/gavin/sandbox/images/rootfs.cpio.xz                           \
   -append "earlycon=pl011,mmio,0x9000000"                                     \
   -device virtio-net-pci,netdev=unet,mac=52:54:00:f1:26:a6                    \
   -netdev user,id=unet,hostfwd=tcp::50959-:22                                 \
   -drive file=/home/gavin/sandbox/images/vm.img,if=none,format=raw,id=nvme0   \
   -device nvme,drive=nvme0,serial=foo                                         \
   -drive file=/home/gavin/sandbox/images/vm1.img,if=none,format=raw,id=nvme1  \
   -device nvme,drive=nvme1,serial=foo1
}

[  160.889802] Unable to handle kernel paging request at virtual address 003fffff7fc00034
[  160.897712] Mem abort info:
[  160.900507]   ESR = 0x96000004
[  160.903550]   EC = 0x25: DABT (current EL), IL = 32 bits
[  160.908848]   SET = 0, FnV = 0
[  160.911896]   EA = 0, S1PTW = 0
[  160.915024] Data abort info:
[  160.917891]   ISV = 0, ISS = 0x00000004
[  160.921722]   CM = 0, WnR = 0
[  160.924678] [003fffff7fc00034] address between user and kernel address ranges
[  160.931808] Internal error: Oops: 96000004 [#1] SMP
[  160.936676] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc rfkill ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib vfat fat ib_umad rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi ipmi_ssif qedr ib_uverbs crct10dif_ce i2c_smbus ghash_ce sha2_ce sha256_arm64 ib_core sha1_ce ipmi_devintf ipmi_msghandler thunderx2_pmu ip_tables xfs libcrc32c sg ast i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper qede ttm mpt3sas qed drm raid_class e1000e scsi_transport_sas crc8 gpio_xlp i2c_xlp9xx dm_mirror dm_region_hash dm_log dm_mod
[  161.007565] CPU: 222 PID: 4559 Comm: qemu-system-aar Not tainted 5.9.0-rc3-gavin+ #4
[  161.015293] Hardware name: Default string MT91-FS1/MT91-FS1, BIOS 28m 12/14/2019
[  161.022676] pstate: 60400009 (nZCv daif +PAN -UAO BTYPE=--)
[  161.028250] pc : __free_pages+0x24/0x60
[  161.032074] lr : free_pages.part.102+0x2c/0x38
[  161.036504] sp : fffffe0031b2f8a0
[  161.039805] x29: fffffe0031b2f8a0 x28: 0000000040000000
[  161.045104] x27: fffffe0031b2f9c8 x26: 0000000000000007
[  161.050402] x25: 0000000000000003 x24: fffffe0010f16000
[  161.055700] x23: 0000000020000000 x22: 0000000040000000
[  161.060998] x21: 0000000000000002 x20: 0000000060000000
[  161.066296] x19: fffffc0f1b050010 x18: 0000000000000000
[  161.071595] x17: 0000000000000000 x16: 0000000000000000
[  161.076893] x15: 0000000000000000 x14: 0000000000000000
[  161.082191] x13: 0000000000000000 x12: 0000000000000001
[  161.087489] x11: 0000000000000003 x10: 0000000000000002
[  161.092787] x9 : fffffe001035fca4 x8 : 0000000000000007
[  161.098085] x7 : 00000000fffffff3 x6 : fffffe0010126370
[  161.103383] x5 : fffffe0031b2f9e8 x4 : 0000040080000000
[  161.108681] x3 : 003fffff7fc00034 x2 : 00000000ffffffff
[  161.113979] x1 : 0000000000000000 x0 : 003fffff7fc00000
[  161.119277] Call trace:
[  161.121713]  __free_pages+0x24/0x60
[  161.125189]  free_pages.part.102+0x2c/0x38
[  161.129272]  free_pages+0x1c/0x28
[  161.132586]  stage2_map_walker+0xbc/0x218
[  161.136584]  __kvm_pgtable_walk+0xec/0x1c8
[  161.140667]  _kvm_pgtable_walk+0xa4/0xe0
[  161.144578]  kvm_pgtable_stage2_map+0xa4/0x118
[  161.149022]  kvm_handle_guest_abort+0x48c/0xa08
[  161.153543]  handle_exit+0x134/0x198
[  161.157107]  kvm_arch_vcpu_ioctl_run+0x4f0/0x880
[  161.161721]  kvm_vcpu_ioctl+0x3a8/0x808
[  161.165546]  __arm64_sys_ioctl+0x1dc/0xcf8
[  161.169642]  do_el0_svc+0xf4/0x1b8
[  161.173039]  el0_sync_handler+0xf8/0x124
[  161.176949]  el0_sync+0x140/0x180
[  161.180254] Code: d503201f 9100d003 52800022 4b0203e2 (b8e20064)
[  161.186408] ---[ end trace d0b1b117875f8fcd ]---
[  161.191012] Kernel panic - not syncing: Fatal exception
[  161.196247] SMP: stopping secondary CPUs
[  161.200206] Kernel Offset: 0xc0000 from 0xfffffe0010000000
[  161.205677] PHYS_OFFSET: 0x80000000
[  161.209154] CPU features: 0x0046002,22800c38
[  161.213410] Memory Limit: none
[  161.216474] ---[ end Kernel panic - not syncing: Fatal exception ]---


FAIL[2]
================================================================
start_vm_aarch64_hugetlbfs() {
   echo 4096 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

   /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                     \
   --enable-kvm -machine virt,gic-version=host                                 \
   -cpu host -smp 8,sockets=8,cores=1,threads=1                                \
   -m 4G -mem-prealloc -mem-path /dev/hugepages                                \
   -monitor none -serial mon:stdio -nographic -s                               \
   -bios /home/gavin/sandbox/qemu.main/pc-bios/edk2-aarch64-code.fd            \
   -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image               \
   -initrd /home/gavin/sandbox/images/rootfs.cpio.xz                           \
   -append "earlycon=pl011,mmio,0x9000000"                                     \
   -device virtio-net-pci,netdev=unet,mac=52:54:00:f1:26:a6                    \
   -netdev user,id=unet,hostfwd=tcp::50959-:22                                 \
   -drive file=/home/gavin/sandbox/images/vm.img,if=none,format=raw,id=nvme0   \
   -device nvme,drive=nvme0,serial=foo                                         \
   -drive file=/home/gavin/sandbox/images/vm1.img,if=none,format=raw,id=nvme1  \
   -device nvme,drive=nvme1,serial=foo1
}

[  666.278391] Unable to handle kernel paging request at virtual address 03fffffefde00034
[  666.286304] Mem abort info:
[  666.289086]   ESR = 0x96000004
[  666.292142]   EC = 0x25: DABT (current EL), IL = 32 bits
[  666.297440]   SET = 0, FnV = 0
[  666.300481]   EA = 0, S1PTW = 0
[  666.303616] Data abort info:
[  666.306484]   ISV = 0, ISS = 0x00000004
[  666.310306]   CM = 0, WnR = 0
[  666.313269] [03fffffefde00034] address between user and kernel address ranges
[  666.320393] Internal error: Oops: 96000004 [#1] SMP
[  666.325259] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc rfkill ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ib_umad vfat fat rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi qedr ib_uverbs ipmi_ssif ib_core crct10dif_ce i2c_smbus ghash_ce sha2_ce sha256_arm64 sha1_ce ipmi_devintf ipmi_msghandler thunderx2_pmu ip_tables xfs libcrc32c sg ast i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper qede ttm qed mpt3sas drm e1000e raid_class crc8 scsi_transport_sas gpio_xlp i2c_xlp9xx dm_mirror dm_region_hash dm_log dm_mod
[  666.396150] CPU: 168 PID: 42112 Comm: qemu-system-aar Not tainted 5.9.0-rc3-gavin+ #5
[  666.403965] Hardware name: Default string MT91-FS1/MT91-FS1, BIOS 28m 12/14/2019
[  666.411348] pstate: 60400009 (nZCv daif +PAN -UAO BTYPE=--)
[  666.416922] pc : __free_pages+0x24/0x60
[  666.420746] lr : free_pages.part.102+0x2c/0x38
[  666.425176] sp : ffffffc024a23840
[  666.428477] x29: ffffffc024a23840 x28: 0000000040000000
[  666.433776] x27: ffffffc024a239c8 x26: 0000000000000007
[  666.439074] x25: 0000000000000003 x24: ffffffc010ec0000
[  666.444373] x23: 0000000000200000 x22: 0000000040200000
[  666.449671] x21: 0000000040000000 x20: ffffff8f34576000
[  666.454969] x19: 0000000000000002 x18: 0000000000000000
[  666.460267] x17: 0000000000000000 x16: 0000000000000000
[  666.465565] x15: 0000000000000000 x14: 0000000000000000
[  666.470863] x13: 0000000000000000 x12: 0000000000000001
[  666.476161] x11: 0000000000000003 x10: 0000000000000002
[  666.481459] x9 : ffffffc0103522f4 x8 : 0000000000000007
[  666.486757] x7 : ffffffc0249960f8 x6 : ffffffc0101162f8
[  666.492055] x5 : ffffffc024a239e8 x4 : 0000000000000004
[  666.497353] x3 : 03fffffefde00034 x2 : 00000000ffffffff
[  666.502651] x1 : 0000000000000000 x0 : 03fffffefde00000
[  666.507950] Call trace:
[  666.510385]  __free_pages+0x24/0x60
[  666.513861]  free_pages.part.102+0x2c/0x38
[  666.517945]  free_pages+0x1c/0x28
[  666.521260]  stage2_map_walker+0xb0/0x208
[  666.525257]  __kvm_pgtable_walk+0xe0/0x1b8
[  666.529340]  __kvm_pgtable_walk+0xb8/0x1b8
[  666.533424]  _kvm_pgtable_walk+0xa4/0xe0
[  666.537334]  kvm_pgtable_stage2_map+0xa0/0x118
[  666.541779]  kvm_handle_guest_abort+0x48c/0xa38
[  666.546300]  handle_exit+0x134/0x198
[  666.549864]  kvm_arch_vcpu_ioctl_run+0x4f0/0x880
[  666.554479]  kvm_vcpu_ioctl+0x3a8/0x808
[  666.558304]  __arm64_sys_ioctl+0x1dc/0xcf8
[  666.562402]  do_el0_svc+0xf4/0x1b8
[  666.565799]  el0_sync_handler+0xf8/0x124
[  666.569709]  el0_sync+0x140/0x180
[  666.573014] Code: d503201f 9100d003 52800022 4b0203e2 (b8e20064)
[  666.579197] ---[ end trace 52b60e2f408396b6 ]---
[  666.583801] Kernel panic - not syncing: Fatal exception
[  666.589035] SMP: stopping secondary CPUs
[  666.592996] Kernel Offset: 0xb0000 from 0xffffffc010000000
[  666.598467] PHYS_OFFSET: 0x80000000
[  666.601944] CPU features: 0x0046002,22800c38
[  666.606200] Memory Limit: none
[  666.609264] ---[ end Kernel panic - not syncing: Fatal exception ]---

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@xxxxxxxxxxxxxxxxxxxxx
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm



[Index of Archives]     [Linux KVM]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux