On 7/10/23 01:17, Pekka Paalanen wrote:
On Fri, 7 Jul 2023 13:54:21 -0700
Michael Banack <banackm@xxxxxxxxxx> wrote:
On 7/7/23 01:38, Pekka Paalanen wrote:
...
The question of which input device corresponds to which cursor plane
might be good to answer too. I presume the VM runner is configured to
expose exactly one of each, so there can be only one association?
As far as I know, all of the VM consoles are written as though they
taking the place of what would the the physical monitors and input
devices on a native machine. So they assume that there is one user,
sitting in front of one console, and all monitors/input devices are
being used by that user.
Ok, but having a single user does not mean that there cannot be
multiple independent pointers, especially on Wayland. The same with
keyboards.
True, and if the userspace is doing anything complicated here, the
hypervisor has to be responsible for ensuring that whatever it's doing
works with that, or else this system won't work. I don't know that the
kernel is in a good position to police that.
What do you mean by policing here?
Isn't it the hypervisor that determines what virtual input devices will
be available to the guest OS? Therefore, the hypervisor is in a
position to expose exactly one pointer device and exactly one
cursor plane to guest OS which means the guest OS cannot get the
association wrong. If that's the general and expected hypervisor
policy, then there is no need to design explicit device association in
the guest kernel UAPI. If so, I'd like it to be mentioned in the kernel
docs, too.
I'm not entirely sure how to fit what you're calling a "pointer" into my
mental model of what the hypervisor is doing...
My definition: A pointer is a pointing input device that requires a
cursor image to be drawn at the right location for it to be usable.
Right, but normal desktops (and our consoles) expect multiple input
devices to feed into a single cursor. So the connection between the
on-screen cursor and the corresponding input-devices is not clear to me
when you start talking about multiple pointers, even without any
hypervisors in the picture.
The connection is simple: there is an independent on-screen cursor for
each logical pointer. How that cursor is drawn is irrelevant to the end
user, and Wayland compositors (a type of a display server) will use any
means necessary to draw it.
Each logical pointer has one cursor that is independent from all other
logical pointers. Each logical pointer can have any number of input
devices controlling it. The assignments are decided by the userspace
and implemented in a display (window system) server.
This has been ingrained into the fundamental design of Wayland, even if
the feature is rarely used in practise. The window system may expose
multiple independent pointers to applications, and each pointer may also
interact with the same window simultaneously. This naturally leads to
the question "which cursor goes with which input device?", and the
kernel, or anything below it, does not know that if there are multiple
possibilities.
Right, but the whole notion of a "pointer" is also not a kernel concept
as far as I know? The kernel only knows that it's requested to display
a particular set of screen contents on an particular CRTC output. What
we're asking for here is for the new HOTSPOT property to allow the final
desired screen contents to have some opt-in flexibility on how it gets
displayed to meet the needs of the para-virtualized drivers and their
use of cursors.
I think trying to extend that beyond the display side to include the
mouse/input semantics when those are not already part of the kernel
interface isn't going to work well.
So I guess I'm not clear on what kind of usermode<=>kernel contract you
want here if the kernel isn't what's owning the translation between the
mouse input and the cursor position. The hypervisor awkwardly has to
straddle both the input/graphics domain, and we do so by making
assumptions about how the user desktop is going to behave.
Correct. In order to reduce that awkwardness, I encourage you to write
down the expectations and requirements in this new Linux UAPI (the KMS
cursor place hotspot property). Then you can be much more confident on
how a random Linux desktop will behave.
It will also help the reviewers here to understand what the new UAPI is
and how it is supposed to work.
The cursor hotspot is I think fairly straightforward, as far as what
that means for how hypervisor clients are expected to draw the mouse,
and Zack's working on that part.
My point was that how the hypervisor then sends input is sort of outside
the scope of the graphics portion here, and I think probably outside the
current Linux UAPI entirely (unless there's some other input/topology
system somewhere else I'm not familiar with).
I would not say that the hotspot property is in any way obvious. I've
spent my whole Wayland and compositor developer career of over 10 years
never seeing the need of the kernel knowing the hotspot myself, because
I never use VMWare like tools.
Sorry, I meant straight-forward as in well-defined, not as in obvious.
It should definitely be documented.
We can just clearly specify the graphics contract here in a way that is
trickier for the mouse/input parts.
You cannot describe why hotspot property is needed or how it works
without explaining the input side. The hotspot property is actually
really weird, because its existence requires combining the input system
with the graphics system for it to make sense. The input system used to
be out of scope indeed, but this addition forces it in scope. This is
the first time that I know of when the kernel or components below the
kernel needs to know, hence there is no existing infrastructure in
Linux to record that topology or anything.
I think the problem here is going to be that each of the userspaces and
para-virtual drivers handle this differently...
They all have their own mechanisms to pass the topology/input-mapping up
to the hypervisor (either custom IOCTLs on their graphics driver, or
else backdoor-style from userspace), and how that works at the userspace
layer (ie X11/Wayland) is differently.
(Sidetrack: for ultra-fast displays, say 1000 Hz refresh for seamless
hand-eye coordination, it could make very much sense for userspace to
off-load arbitrary KMS plane positioning through the kernel into
hardware in general, tying the plane position to some input device
position temporarily. It might make some sense even with today's
consumer hardware. So I don't think the concept is fundamentally
limited to paravirtualized environments. But going there would need a
lot more work than I am willing to suggest to VMWare to tackle to just
make their own products better.)
Sure, but to do that we would need to standardize how the input mapping
happens, and create standard kernel interfaces for the topology
information, and relative mouse acceleration curves, etc. That's the
piece that I'm saying isn't standardized enough now to be able to spell
that out in documentation now, because there is no standard way of
handling all that.
As a hand-waiving, "this is kind of how it works", it's useful to
document. But if you're pushing for a hard "this is how the Linux
kernel handles mouse input", then I don't think any of these components
are ready to commit to the mouse/input handling as a Linux UAPI.
So I guess I would vote for trying to include something to that effect
as context or explanation, but not try to strictly define how that works?
Yes, exactly.
Okay, if we can keep the mouse/input stuff on the fuzzy side then I
think we're on the same page.
Thanks!
--Michael Banack
