Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On 2021/12/14 00:49, Marc Zyngier wrote:
On Mon, 13 Dec 2021 16:06:14 +0000,
Peter Maydell <peter.maydell@xxxxxxxxxx> wrote:

KVM on big.little setups is a kernel-level question really; I've
cc'd the kvmarm list.

Thanks Peter for throwing us under the big-little bus! ;-)

On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote:

On 2021/12/13 21:17, Michal Prívozník wrote:
On 12/11/21 02:58, Qu Wenruo wrote:

Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
with upstream kernels.

For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
brings quite some troubles for VMs.

In short, without proper cpuset to bind the VM to either all A72 cores
or all A55 cores, the VM will mostly fail to boot.

s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
though of it makes me sick).

Currently the working xml is:

    <vcpu placement='static' cpuset='4-5'>2</vcpu>
    <cpu mode='host-passthrough' check='none'/>

But even with vcpupin, pinning each vcpu to each physical core, VM will
mostly fail to start up due to vcpu initialization failed with -EINVAL.

Disclaimer: I know nothing about libvirt (and no, I don't want to
know! ;-).

However, for things to be reliable, you need to taskset the whole QEMU
process to the CPU type you intend to use.

Yep, that's what I'm doing.

That's because, AFAICT,
QEMU will snapshot the system registers outside of the vcpu threads,
and attempt to use the result to configure the actual vcpu threads. If
they happen to run on different CPU types, the sysregs will differ in
incompatible ways and an error will be returned. This may or may not
be a bug, I don't know (I see it as a feature).

Then this brings another question.

If we can pin each vCPU to each physical core (both little and big),
then as long as the registers are per-vCPU based, it should be able to
pass both big and little cores to the VM.

Yeah, I totally understand this screw up the scheduling, but that's at
least what (some insane) users want (just like me).

If you are annoyed with this behaviour, you can always use a different
VMM that won't care about such difference (crosvm or kvmtool, to name
a few).

Sounds pretty interesting, a new world but without libvirt...

However, the guest will be able to observe the migration from
one cpu type to another. This may or may not affect your guest's

Not sure if it's possible to pin each vCPU thread to each core, but let
me try.

I personally find the QEMU behaviour reasonable. KVM/arm64 make little
effort to support BL virtualisation as design choice (I value my
sanity), and userspace is still in control of the placement.

This brings a problem, in theory RK3399 SoC should out-perform BCM2711
in multi-core performance, but if a VM can only be bind to either A72 or
A55 cores, then the performance is no longer competitive against
BCM2711, wasting the PCIE 2.0 x4 capacity.

Vote with your money. If you too think that BL systems are utter crap,
do not buy them! Or treat them as 'two systems in one', which is what
I do. From that angle, this is of great value! ;-)

I guess I'm setting my expectation too high for rk3399, just seeing its
multi-thread perf beating RPI4 and has better IO doesn't mean it's a
perfect fit for VM.

Hopes rk3588 could change it.

For now I guess overclocking the big core to 2.2G is what I can do to
grab more performance from the board.

Thanks for your detailed reason and new advices!

I guess with projects like Asahi Linux making progress, there will be
more and more such problems.

Well, not more than any other big-little system. They suffer from
similar issues, plus those resulting from not fully implementing the
ARM architecture. They are however more consistent in their feature
set than the ARM implementations ever were.

Any clue on how to properly pass all physical CPU cores to VM for
little.BIG setup?

I have never met big.LITTLE but my understanding was that those big
cores are compatible with little ones and the only difference is that
the big ones are shut off if there's no demand (to save energy) leaving
only the little ones running.

No. They are all notionally running. It is the scheduler that places
tasks (such as a vcpu) on a 'convenient' core, where 'convenient'
depends on the scheduling policy.



[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux