On Tue, Dec 18, 2018 at 01:25:32PM +0000, Andrew Murray wrote:
> On Tue, Dec 18, 2018 at 01:02:26PM +0100, Christoffer Dall wrote:
> > On Wed, Dec 12, 2018 at 10:29:32AM +0000, Andrew Murray wrote:
> > > Add support for the :G and :H attributes in perf by handling the
> > > exclude_host/exclude_guest event attributes.
> > >
> > > We notify KVM of counters that we wish to be enabled or disabled on
> > > guest entry/exit and thus defer from starting or stopping :G events
> > > as per the events exclude_host attribute.
> > >
> > > With both VHE and non-VHE we switch the counters between host/guest
> > > at EL2. We are able to eliminate counters counting host events on
> > > the boundaries of guest entry/exit when using :G by filtering out
> > > EL2 for exclude_host. However when using :H unless exclude_hv is set
> > > on non-VHE then there is a small blackout window at the guest
> > > entry/exit where host events are not captured.
> > >
> > > Signed-off-by: Andrew Murray <andrew.murray@xxxxxxx>
> > > ---
> > > arch/arm64/kernel/perf_event.c | 51 ++++++++++++++++++++++++++++++++++++------
> > > 1 file changed, 44 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/arch/arm64/kernel/perf_event.c b/arch/arm64/kernel/perf_event.c
> > > index de564ae..4a3c73d 100644
> > > --- a/arch/arm64/kernel/perf_event.c
> > > +++ b/arch/arm64/kernel/perf_event.c
> > > @@ -26,6 +26,7 @@
> > >
> > > #include <linux/acpi.h>
> > > #include <linux/clocksource.h>
> > > +#include <linux/kvm_host.h>
> > > #include <linux/of.h>
> > > #include <linux/perf/arm_pmu.h>
> > > #include <linux/platform_device.h>
> > > @@ -647,11 +648,26 @@ static inline int armv8pmu_enable_counter(int idx)
> > >
> > > static inline void armv8pmu_enable_event_counter(struct perf_event *event)
> > > {
> > > + struct perf_event_attr *attr = &event->attr;
> > > int idx = event->hw.idx;
> > > + int flags = 0;
> > > + u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx));
> > >
> > > - armv8pmu_enable_counter(idx);
> > > if (armv8pmu_event_is_chained(event))
> > > - armv8pmu_enable_counter(idx - 1);
> > > + counter_bits |= BIT(ARMV8_IDX_TO_COUNTER(idx - 1));
> > > +
> > > + if (!attr->exclude_host)
> > > + flags |= KVM_PMU_EVENTS_HOST;
> > > + if (!attr->exclude_guest)
> > > + flags |= KVM_PMU_EVENTS_GUEST;
> > > +
> > > + kvm_set_pmu_events(counter_bits, flags);
> > > +
> > > + if (!attr->exclude_host) {
> > > + armv8pmu_enable_counter(idx);
> > > + if (armv8pmu_event_is_chained(event))
> > > + armv8pmu_enable_counter(idx - 1);
> > > + }
> > > }
> > >
> > > static inline int armv8pmu_disable_counter(int idx)
> > > @@ -664,11 +680,20 @@ static inline int armv8pmu_disable_counter(int idx)
> > > static inline void armv8pmu_disable_event_counter(struct perf_event *event)
> > > {
> > > struct hw_perf_event *hwc = &event->hw;
> > > + struct perf_event_attr *attr = &event->attr;
> > > int idx = hwc->idx;
> > > + u32 counter_bits = BIT(ARMV8_IDX_TO_COUNTER(idx));
> > >
> > > if (armv8pmu_event_is_chained(event))
> > > - armv8pmu_disable_counter(idx - 1);
> > > - armv8pmu_disable_counter(idx);
> > > + counter_bits |= BIT(ARMV8_IDX_TO_COUNTER(idx - 1));
> > > +
> > > + kvm_clr_pmu_events(counter_bits);
> > > +
> > > + if (!attr->exclude_host) {
> > > + if (armv8pmu_event_is_chained(event))
> > > + armv8pmu_disable_counter(idx - 1);
> > > + armv8pmu_disable_counter(idx);
> > > + }
> > > }
> > >
> > > static inline int armv8pmu_enable_intens(int idx)
> > > @@ -943,16 +968,25 @@ static int armv8pmu_set_event_filter(struct hw_perf_event *event,
> > > * Therefore we ignore exclude_hv in this configuration, since
> > > * there's no hypervisor to sample anyway. This is consistent
> > > * with other architectures (x86 and Power).
> > > + *
> > > + * To eliminate counting host events on the boundaries of
> > > + * guest entry/exit we ensure EL2 is not included in hyp mode
> > > + * with !exclude_host.
> > > */
> > > if (is_kernel_in_hyp_mode()) {
> > > - if (!attr->exclude_kernel)
> > > + if (!attr->exclude_kernel && !attr->exclude_host)
> > > config_base |= ARMV8_PMU_INCLUDE_EL2;
> > > } else {
> > > - if (attr->exclude_kernel)
> > > - config_base |= ARMV8_PMU_EXCLUDE_EL1;
> > > if (!attr->exclude_hv)
> > > config_base |= ARMV8_PMU_INCLUDE_EL2;
> >
> > I'm not sure about the current use of exclude_hv here. The comment says
> > it's consistent with other architectures, but I can't find an example to
> > confirm this, and I don't think we have a comparable thing to the split
> > of the hypervisor between EL1 and EL2 we have on non-VHE.
> >
> > Joerg told me the semantics were designed to be:
> >
> > exclude_hv: When running as a guest, stop counting events when
> > the HV runs.
>
> Can the definition of "guest" here refer to both type 1 and type 2
> hypervisor guests? Or do we assume type 1 only?
>
A guest is a guest. Linux can run as a guest under a hypervisor with a
type 1 or type 2 design, doesn't matter for this conversation.
> >
> > exclude_host: When Linux runs as a HV itself, only count events
> > while a guest is running.
> >
> > exclude_guest: When Linux runs as a HV, only count events when
> > running in host mode.
> >
> > (But tools/perf/design.txt does not really confirm this).
> >
> > On arm64 that would mean:
> >
> > exclude_hv: As a host, no effect.
> > As a guest, set the counter to include EL2 for a
> > hypervisor to emulate.
[...]
>
> Though more correctly we should count EL2 *and EL1* events whilst pinned
> to the KVM task and whilst running outside of the guest. This then
> covers both !VHE and VHE and allows for fair comparasion between !VHE
> and VHE systems.
Yes, if the guest has cleared exclude_hv and if we can properly detect
that from the hypervisor.
>
> This then gives us the unique benefit of the type 2 host being able to
> examine the hypervisor overhead of its individual guests.
Not sure I understand this part.
>
> The only issue here is that the type 2 host wouldn't be able to examine
> the HV overhead of all its guests across the system as you wouldn't be
> able to rely on the perf task pinning to distinguish between EL1 from
> host and EL1 from guests in a !VHE system. I'm not sure the best way
> to overcome this limitation.
Why can't you disable EL1 counting whilst running in the host, and
enable EL1 counting whilst running in the guest?
> >
> > exclude_host: As a guest, has no effect.
> > Don't count EL1 host or EL2, but count EL1 guest
> > by enabling EL1 counting at EL2 when entering a
> > guest, and disabling EL1 counting when returning
> > from a guest.
> >
> > exclude_guest: As a guest, has no effect. As a host, disable
> > EL1 counting at EL2 when entering a guest.
> >
> > Not sure if we break anything by changing the behavior on arm64 now, but
> > I really doubt that being able to exclude an arbitrary part (the one tha
> > happens to run in EL2 on non-VHE systems) is meaningful, and the fact
> > that behavior and semantics change depending on the version of the
> > underlying CPU is not great, if what you care about is understanding the
> > system's performance.
>
> This is a bit strange. It's arbitary as it only represents a bit of the
> HV overhead - this is solved though by counting the whole overhead (EL1
> and EL2 instead (but only counting outside the guest and pinned to the
> guest tasks).
Not sure I understand your point here?
>
> >
> > Thoughts?
> >
>
> Though if I've understood you correctly, you're suggesting that the only
> time we count EL2 is when exclude_hv is not set on the immediate guest
> of a type 1 hypervisor?
>
No, I didn't say anything about a type 1 or type 2 hypervisor, and I
think that distinction is completely irrelevant to the discussion at
hand. I also don't know what an immediate guest is -- is there any
other kind?
I don't think exclude_hv, exclude_host, and exclude_guest are directly
tied to a single CPU mode. The only 'modes' you need to consider for
Linux are 'guest' and 'host' when Linux can run VMs and, 'self' and
'hypervisor' when Linux is a guest.
When Linux can run VMs, you count EL2 events when exclude_host is not
set.
(When Linux is a guest, and you set/clear exclude_hv, for this to work,
you need some way of informing your hypervisor that you want to know
about events happening in the hypervisor. This could be a PV interface,
or maybe this can work by the guest setting/clearing the NSH bit in its
virtual PMU registers, which then amusingly can get translated into
actually counting in EL1/EL2 (non-VHE) or EL2 (VHE) by KVM's PMU
emulation code. The method used is specific to the hypervisor used, but
not specific to whether the hypervisor is type-1 or type-2.)
Thanks,
Christoffer
_______________________________________________
kvmarm mailing list
kvmarm@xxxxxxxxxxxxxxxxxxxxx
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
