On Tue, Aug 07, 2018 at 12:23:45PM +0100, Dave Martin wrote: > On Mon, Aug 06, 2018 at 03:41:33PM +0200, Christoffer Dall wrote: > > On Thu, Jul 26, 2018 at 02:18:02PM +0100, Dave Martin wrote: > > > On Wed, Jul 25, 2018 at 06:52:56PM +0200, Andrew Jones wrote: > > > > On Wed, Jul 25, 2018 at 04:27:49PM +0100, Dave Martin wrote: > > > > > On Thu, Jul 19, 2018 at 04:59:21PM +0200, Andrew Jones wrote: > > > > > > On Thu, Jun 21, 2018 at 03:57:40PM +0100, Dave Martin wrote: > > > > > > > - /* > > > > > > > - * For now, we don't return any features. > > > > > > > - * In future, we might use features to return target > > > > > > > - * specific features available for the preferred > > > > > > > - * target type. > > > > > > > - */ > > > > > > > + /* KVM_ARM_VCPU_SVE understood by KVM_VCPU_INIT */ > > > > > > > + init->features[0] = 1 << KVM_ARM_VCPU_SVE; > > > > > > > + > > > > > > > > > > > > We shouldn't need to do this. The "preferred" target type isn't defined > > > > > > well (that I know of), but IMO it should probably be the target that > > > > > > best matches the host, minus optional features. The best base target. We > > > > > > may use these features to convey that the preferred target should enable > > > > > > some optional feature if that feature is necessary to workaround a bug, > > > > > > i.e. using the "feature" bit as an erratum bit someday, but that'd be > > > > > > quite a debatable use, so maybe not even that. Most likely we'll never > > > > > > need to add features here. > > > > > > > > > > init->features[] has no semantics yet so we can define it how we like, > > > > > but I agree that the way I use it here is not necessarily the most > > > > > natural. > > > > > > > > > > OTOH, we cannot use features[] for "mandatory" features like erratum > > > > > workarounds, because current userspace just ignores these bits. > > > > > > > > It would have to learn to look here if that's how we started using it, > > > > but it'd be better to invent something else that wouldn't appear as > > > > abusive if we're going to teach userspace new stuff anyway. > > > > > > > > > > > > > > Rather, these bits would be for features that are considered beneficial > > > > > but must be off by default (due to incompatibility risks across nodes, > > > > > or due to ABI impacts). Just blindly using the preferred target > > > > > already risks configuring a vcpu that won't work across all nodes in > > > > > your cluster. > > > > > > > > KVM usually advertises optional features through capabilities. A device > > > > (vcpu device, in this case) ioctl can also be used to check for feature > > > > availability. > > > > > > > > > > > > > > So I'm not convinced that there is any useful interpretation of > > > > > features[] unless we interpret it as suggested in this patch. > > > > > > > > > > Can you elaborate why you think it should be used with a more > > > > > concrete example? > > > > > > > > I'm advocating that it *not* be used here. I think it should be used > > > > like the PMU feature uses it - and the PMU feature doesn't set a bit > > > > here. > > > > > > > > > > > > > > > That said, I think defining the feature bit makes sense. ATM, I'm feeling > > > > > > like we'll want to model the user interface for SVE like PMU (using VCPU > > > > > > device ioctls). > > > > > > > > > > Some people expressed concerns about the ioctls becoming order-sensitive. > > > > > > > > > > In the SVE case we don't want people enabling/disabling/reconfiguring > > > > > "silicon" features like SVE after the vcpu starts executing. > > > > > > > > > > We will need an extra ioctl() for configuring the allowed SVE vector > > > > > lengths though. I don't see a way around that. So maybe we have to > > > > > solve the ordering problem anyway. > > > > > > > > Yes, that's why I'm thinking that the vcpu device ioctls is probably the > > > > right way to go. The SVE group can have its own "finalize" request that > > > > allows all other SVE ioctls to be in any order prior to it. > > > > > > > > > > > > > > > > > > > By current approach (not in this series) was to have VCPU_INIT return > > > > > -EINPROGRESS or similar if SVE is enabled in features[]: this indicates > > > > > that certain setup ioctls are required before the vcpu can run. > > > > > > > > > > This may be overkill / not the best approach though. I can look at > > > > > vcpu device ioctls as an alternative. > > > > > > > > With a "finalize" attribute if SVE isn't finalized by VCPU_INIT or > > > > KVM_RUN time, then SVE just won't be enabled for that VCPU. > > > > > > So I suppose we could do something like this: > > > > > > * Advertise SVE availability through a vcpu device capability (I need > > > to check how that works). > > > > > > * SVE-aware userspace that understands SVE can do the relevant > > > vcpu device ioctls to configure SVE and turn it on: these are only > > > permitted before the vcpu runs. We might require an explicit > > > "finish SVE setup" ioctl to be issued before the vcpu can run. > > > > > > * Finally, the vcpu is set running by userspace as normal. > > > > > > Marc or Christoffer was objecting to me previously that this may be an > > > abuse of vcpu device ioctls, because SVE is a CPU feature rather than a > > > device. I guess it depends on how you define "device" -- I'm not sure > > > where to draw the line. > > > > I initially advocated for a VCPU device ioctl as well, because it's a > > less crowded number space that gives you more flexibility. Marc did > > have a strong point that vcpu *devices* implies something else than > > features though. > > > > I think you (a) definitely want to announce SVE support via a > > capability, and (b) only set the preferred target flag if enabling SVE > > *generally* gives you a VM more like the real hardware with similar > > performance on some system. > > > > I'm personally fine with both feature flags and vcpu device ioctls. If > > using vcpu device ioctls gives you an obvious way to set attributes > > relating to SVE, e.g. the vector length, then I think that's a strong > > argument for that approach. > > There is another option I'm tending towards, which is simply to have > a "set vector lengths" ioctl (whether presented as a vcpu device > ioctl or a random arch ioctl). Someone complained once about adding too many arch ioctls because there is a limited number space for doing so, but I'm not sure if that was and still a valid concern. > > If that ioctl() fails then SVE support is not available. > > If it succeeds, it will update its arguments to indicate which > vector lengths are enabled (if different). > > Old userspace, or userspace that doesn't want to use SVE, would > not use this ioctl at all. > > It would also do no harm additionally to advertise this as a > capability, though I wonder whether it's necessary to do so (?) > It is customary to expose features via capabilities. I have a vague recollection that tools like libvirt negotiate capabilities across systems and would need more plumbing to discover features by probing an ioctl instead. Thanks, -Christoffer _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
