Hi Sean,
On Wed, Apr 10, 2019 at 1:29 AM Sean O'Brien <seobrien@xxxxxxxxxxxx> wrote:
>
> Hello,
>
> I'm currently working on designing an interface for controlling "forcepads";
> that is, touchpads with force sensors and haptic actuators. Below is my
> proposal for the protocol at both the userspace and HID interfaces. I would
> appreciate any feedback you might have.
Thanks a lot for starting this discussion. This is indeed an
interesting topic, and I had a lengthy chat with Peter this morning :)
First, I'd like to get a small clarification.
In the proposal, you are making reference to both the HID side (so the
hardware protocol) and the kernel side (the client protocol).
I read it that you are trying to build two things:
- the HID design proposal, aimed at hardware makers
- the kernel behavior on how to use it
If that is the case, I am glad that you can put some pressure to force
a sensible protocol to hardware makers :)
Though, I think we should make a clear separation between the 2. We
need to think of it as a whole, but the document should clearly put
the boundaries IMO.
>
> Thank you,
>
> Sean O'Brien
> Chromium OS Touch/Input Team
>
> Background
> ==========
>
> There are multiple independent projects to develop a touchpad with force sensors
> and haptic actuators, instead of a traditional button. These “forcepads” have
> several advantages and potential uses; they allow clicking across the entire
> touchpad surface, adjusting the force requirement for clicks, haptic feedback
> initiated by UI, etc.
>
> Objective
> =========
>
> Develop a standard protocol to allow userspace applications to communicate with
> forcepads, and minimize duplicated code and effort.
>
> Requirements:
> 1. Support UI-initiated haptic feedback.
> 2. Allow userspace to control when button press and button release haptic
> effects are triggered. (Useful when detecting a false click, changing force
> thresholds, or sending context-dependent effects)
> 3. Allow a backward-compatible mode where the forcepad produces button press and
> button release effects autonomously.
> 4. Reveal force sensor readings to userspace applications.
>
> Proposal
> ========
>
> I propose standardized forcepad support in the linux kernel at both the HID and
> userspace interface.
>
> Userspace Interface
> -------------------
>
> Multitouch
> ..........
>
> The linux kernel has a well defined protocol [0] for multitouch devices,
> including touchpads. The protocol defines a field called ABS_MT_PRESSURE as a
> measure of the pressure applied by a contact. Unfortunately, it is not used
> that way in practice- instead it is generally used as an approximate measure of
> contact area. I will distinguish these concepts by calling them “true force” and
> “traditional pressure”.
Well, yes and now. It depends on the underlying technology.
For capacitive sensors, "traditional pressure" if reported through
ABS_MT_PRESSURE is the surface contact area. This is a rough
equivalent to a pressure, assuming we all have somehow flexible
fingers, and roughly the same shape. But for resistive touchscreens,
IIRC we have already access to the "true force".
IIRC, when both ABS_MT_MAJOR and ABS_MT_MINOR are reported, libinput
ignores ABS_MT_PRESSURE. So in a sense, there is no need to add a new
ABS event, this one works quite well.
>
> Instead of using ABS_MT_PRESSURE to report the “traditional pressure,” forcepads
> would send “true force” values in this field. For each contact, this field would
> report the estimated force applied by that contact.
>
> The resolution of ABS_MT_PRESSURE should also be defined and reported, so that
Please define here what units the protocol expects. And please make it
international (g/m2, N/m2 or whatever). Though Newtons might be
difficult to grasp for people.
> userspace consumers can translate to force units. By defining the resolution, we
> also differentiate it from how it is used to report “traditional pressure”,
> where it has no resolution. Userspace consumers will be able to use this to
> detect that this is a forcepad, and treat the pressure field accordingly.
I tend to disagree. It is best in that case to explicitly mark the
forcepad as such by adding a new INPUT_PROP. INPUT_PROP_FORCEPAD would
be nice.
The reason is you don't know for sure that the reported unit is
correct for the existing devices. If you want this to actually matter,
then you can enforce the unit to be correct for a certified device,
and then only export INPUT_PROP_FORCEPAD for those devices (thinking
at the Win8 certification blob for multitouch screens/touchpads).
Note that defining what an INPUT_PROP_FORCEPAD is would be nice. For
example does the user controlled haptic feedback be a requirement.
I would suggested:
- can differentiate between at least 5 fingers
- correct resolution for the X/Y (units and value)
- report correct force per touch, and correct units for them
- follows the MT protocol type B
So for hardware vendors, we would require to follow the MS spec for
input devices in Win8 and Win8.1, and in addition support the Simple
haptic controller HID table, and report correct units for the
pressure.
>
> ABS_PRESSURE may be optionally reported as the total force applied to the
> forcepad.
>
> The device/driver shouldn’t detect button clicks, this is left to the userspace
> gesture library. Accordingly, the driver should not sent BTN_* events to
> userspace in normal operating mode. However it should still report the ability
> to produce such events, for use in autonomous mode.
For backward compatibility, and to be able to debug it properly, you
should keep the BTN_* events emulated in all cases.
The userspace can ignore the events it doesn't want this way, but you
will be able to debug the btn emulations on your current session
without having to kill your compositor.
There shouldn't be much of a head over forwarding those events, as it
will never come alone, and will always be with an other one at least
(pressure being 0 or less).
Also, not sending BTN_TOUCH and BTN_LEFT might give some headaches to
legacy applications.
>
> Haptic Control
> ..............
>
> The force feedback protocol [1] should be used to request predefined effects.
s/request/control/
>
> Typical use of the force feedback protocol requires loading effects to the
> driver by describing the output waveform, and then requesting those effects
> using an ID provided by the driver. We don’t want to describe the output
> waveform explicitly, but instead use a set of predefined IDs for the desired
> effect types. The device/driver would be responsible for having the effects
> loaded and ready to respond to a request for the predefined IDs.
Re-reading through this made my head a little bit clearer.
I think I am starting to see what you are saying, but the proposal
would need to be more precised here.
>
> The force feedback protocol will need to be extended to allow requests for
> predefined IDs. This requires a new feedback effect type:
>
> /**
> * struct ff_predefined_effect
> * @level: strength of the effect
> * @vendor: ID of the vendor who defined the effect
> * @id: ID of the effect among those defined by the vendor
> */
> struct ff_predefined_effect {
> __s16 level;
> __u16 vendor;
> __u16 id;
> }
>
> Vendors can define specifications for the waveforms and assign them IDs. They
> could then be requested using their vendor ID and the waveform ID, as defined in
> the simple haptic HID protocol [2].
>
> To allow a standard way to trigger press and release effects, all forcepads
> should support the WAVEFORM_PRESS and WAVEFORM_RELEASE through this interface.
> Since the standard waveform id namespace doesn’t overlap with the vendor
> waveform id namespace, the vendor id can be ignored for these waveforms.
First of all, I think you should split the HID (device/firmware)
requirements from how they are used in the kernel.
IMO, it's fine to say that only touchpads exporting the "simple haptic
HID protocol [2]" will be supported by this proposal.
And FWIW, the Microsoft Surface Dial already exposes this HID
collection, and can be used to toy around the proposal (just saying).
If you intend to be more generic, we should not refer to this HID
collection but explain how things are going to be in shape in the
kernel.
That being said, there are a few additions/changes I'd like to see.
The way the input_ff protocol works is:
- userspace upload an effect through EVIOCSFF and stores it in an id
- this effect is supposedly stored into the device itself
- userspace can then play the effect whenever it wants by calling the
id in an input_event written to the evdev node
- eventually, userspace can remove the effect with EVIOCRMFF
This doesn't entirely matches the simple haptic HID protocol.
As I read the documentation, the simple haptic HID only defines a set
of waveform types:
WAVEFORM_NONE, WAVEFORM_STOP, WAVEFORM_CLICK,
WAVEFORM_BUZZ_CONTINUOUS, WAVEFORM_RUMBLE_CONTINUOUS, WAVEFORM_PRESS,
and WAVEFORM_RELEASE. It also defines a "vendor" range where vendors
can put any waveform they see fit and define in their haptic device.
So, I think that's why you want to augment the ff protocol by adding
the struct ff_predefined_effect.
After thinking more about that, you are correct, we can not express
those "simple haptic HID" waveforms in the current ff protocol. But I
think the new struct should be focused on HID, not vendors, and
contains the whole effect settings. All the current struct ff_*_effect
also have the period, the magnitude, the offset, etc...
Re-reading through the HID HUTRR, I think we should add Intensity,
Repeat Count, and Duration.
So I would prefer to have a more generic ff_effect:
struct ff_hid_effect {
__u16 hid_usage;
__s16 intensity;
__u16 repeat_count;
__u16 duration;
}
The id is already part of struct ff_effect, so this should allow us to
"upload" a predefined hid effect.
Note that as I read it, we do not want auto-triggered haptic feedback
here. If we do, we would nee to add an other field to the
ff_hid_effect struct.
The next thing I'd like to get some clarifications is how these
effects are used. As I read it, userspace "uploads" the effect, and
uses it, but the HID HUTRR doesn't say that the effects and parameters
are stored in the device itself. It is for the auto triggered ones,
but given that we want to opt for the manual trigger, we should store
it ourselves.
So, the solution we came to this morning, while talking to Peter, was
that the HID driver for a simple haptic HID device would allocate a
virtual device memory to store the effects and the parameters.
This way, we can:
- upload effect WAVEFORM_RELEASE with its parameters in id 0 of the
drvdata of the device
- upload effect WAVEFORM_PRESS with its parameters in id 1 of the
drvdata of the device
- ...
- upload effect WAVEFORM_VENDOR_ZZZ_ZZZ with its parameters in id N of
the drvdata of the device -> userspace will use it while scrolling for
instance
- ...
Then the kernel on BTN_LEFT press can automatically trigger the effect
with id 1 and the one with id 0 on release in the case of the
autonomous mode mentioned below.
To solve the question of knowing which effect should be loaded in
which slot, I think we should rely on a userspace helper (udev?).
We definitively not want the kernel to keep a list of devices to
effects matches, but having a udev database (hwdb and intrinsic?)
would nicely solve the issue as we do not need to update the kernel
for each new device coming in.
>From the kernel driver, we can populate the WAVEFORM_PRESS and
WAVEFORM_RELEASE with some sensible parameters, but userspace should
be allowed to override them.
The advantage of having this virtual memory of device effects, is that
each userspace implementation could use its own matching for effects.
For example, libinput might want to say:
- id 0 -> BTN_LEFT released
- id 1 -> BTN_LEFT pressed
- id 0x1000 -> scrolling up
- id 0x1001 -> scrolling down
- id 0x2042 -> hard press
But chromeOS might use different ids and different meaning for them.
The first 2 ids first should be reserved for the kernel button left
emulation though.
Note that given that the effect memory is 'virtual' in the driver, we
can use any id and we are not limited to a few numbers of them.
An other thing is that we do not need to require WAVEFORM_PRESS and
WAVEFORM_RELEASE. If they are set in the haptic device, the kernel can
set them for us, but if they are not, userspace can always decide to
put one of the other waveforms in ids 0 and 1.
We should mention that not defining these two waveforms will induce a
state were no haptic feedback will be automatically loaded from the
kernel though (userspace will have to do it).
>
> Autonomous mode
> ---------------
>
> In order to facilitate an operating system which cannot handle force feedback,
> the forcepad should start up in “autonomous mode”, meaning it acts as a normal
> touchpad. This means it should perform the press and release haptic feedback
> autonomously at predefined force thresholds, and send the appropriate BTN_*
> events.
I think that if we were to follow my idea of using effect ids 0 and 1
for release/press, we should mention that the kernel will play the
effects stored as these ids.
>
> After verifying that all of the required haptic effects are available through
> the force feedback protocol, the OS can enable host-controlled mode. This could
> be done by writing to a sysfs node “host_feedback_enabled”.
You don't really want a sysfs file here. Because if the input part
generating the haptic feedback crashes, or if you are VT switching and
your evdev node gets revoked, you are left in a system without haptic
feedback for clicks.
So, our idea was to use a new couple of ioctls, EVIOCFFTAKEMASTER and
EVIOCFFRELEASEMASTER (or whatever is more politically admitted), which
the kernel would ref count on each effect id.
So, if the OS decides to take control over the press autonomous haptic
feedback, it will emit the ioctl EVIOCFFTAKEMASTER with the effect id
1. If an other client also wants to control this id, it can also take
the control, and the total count of clients controlling this effect id
is now 2. If a BTN_LEFT is emitted by the device (or emulated by the
kernel), the driver will check the refcount of the effect id, sees
that there is at least one, and will not forward the haptic effect to
the device.
On the contrary, in this case, effect id 0 is not handled by
userspace, and the kernel will control it when a BTN_LEFT release is
emitted.
Note that this ioctl will only have an impact on effect ids 0 and 1.
The benefit of using an ioctl is that is the client closes the fd, or
if the fd gets revoked, the kernel can decrement de usage count and
eventually switch back to the autonomous mode described above.
>
> When the host enters suspend mode, the OS will not be able to respond quickly
> enough to input from the touchpad to tell it to perform haptic feedback, making
> the touchpad feel unresponsive. When the host suspends, the touchpad should
> enter autonomous mode.
This could be achieved in my proposal above by either closing the fd,
or revoking it, so we would get this for free :)
>
> HID Interface
> -------------
So this is the part where we tell ODM what to do. Correct?
>
> Multitouch
> ..........
>
> The HID API for multitouch reports is mostly unchanged except:
>
> 1. The tip pressure field [3] should be used to report “true force” instead of
> “traditional pressure”. The physical unit type, exponent, and limits should
> be reported in the report descriptor for the “true force” field [4].
works for me
> 2. The device will always report it’s button as being unpressed, except in
> autonomous mode, when it will report the button state according to its
> predefined force thresholds.
I think I would prefer having a basic left button emulation that we
can rely on when userspace doesn't know about forcepads.
This would allow us to not have to emulate it in the kernel, but just
forward the data as it comes in.
Also, this would allow the driver to not have to drive the haptic
feedback in the autonomous mode: we just set the auto trigger on
BTN_LEFT usage, and done.
(if we do not enforce the auto triggering, then the driver will also
have to manually send the haptic events on BTN_LEFT 0/1).
If we are giving requirements to ODM, I think we also need:
3. a specific application usage that tells us that the device is a forcepad.
Ideally, I would love to extend the HUT, but we might consider saying
that a forcepad, to be exposed as such, needs to expose a "Simple
Haptic Controller" logical collection alongside the Touch Pad one.
That's what the Surface dial is doing:
- application collection: System Multi-Axis Controller
- logical collection: Puck
- logical collection: Simple Haptic Controller
- logical collection: Durations List
- logical collection: Waveforms List
- endof application collection
So this would make clear what a HID forcepad is.
4. a way to store in the HID device the pressure threshold the device
is supposed to report the click.
We can guess which threshold it is if the unit is correct, but I think
I would rather have the device telling us this. Problem is, I can not
find a good HID usage for that.
5. If a device doesn't meet those criteria above, it will not get
categorized as a FORCEPAD and will not get all the fancy FF effects
from the OS.
This last point is very important as this will prevent us to have to
deal with quirky devices.
Note that MS solved the issue by requiring ODM to actually test their
devices, and then MS issues a blob in a vendor feature that is tested
to enable or not the touchscreen / touchpad gestures.
>
>
> Haptic control
> ..............
>
> The simple haptic HID protocol [2] should be used.
s/should/must/
>
> The following waveforms should be supported:
>
> | WAVEFORM_NONE | Implicit waveforms required by protocol |
> | WAVEFORM_STOP | |
> | ------------------------ | ------------------------------------------------- |
> | WAVEFORM_PRESS | To be used in autonomous mode and host-controlled |
> | WAVEFORM_RELEASE | mode to simulate button press and release. |
I would not absolutely require those 2. If they are present, the
kernel (driver) should set them as the default feedback (in the
autonomous mode), but if they aren't, it's not a big deal, the
userspace will have to quirk the device.
> | ------------------------ | ------------------------------------------------- |
> | Vendor-defined waveforms | Optional waveforms to be used in host-controlled |
> | | mode, subject to vendor specification. |
I think we should just say that the device is free to implement all
the waveforms it wants as per HUTRR 63b.
>
> All waveforms will have an associated duration; continuous waveforms will not be
> supported.
>
> Only manual triggering will be supported through this interface. Autonomous
> triggering of waveforms is enabled by putting the device in autonomous mode.
This part is ambiguous. The Auto triggering on the HUTRR 63b says that
when the device emits a specific HID usage, it will also emit the
matching haptic effect.
While here, you want the kernel to driver the button press/release
(unless I am reading it wrong).
Also, I think you are mixing here the HID requirements and the
exported kernel API. I think there is nothing wrong to comply with the
HUTRR 63b. We can use the auto-trigger functionality in the autonomous
mode described above. The point is that we do not want to export that
to the userspace.
>
> Retriggering (queueing multiple triggers of the same waveform) is not supported.
This will simplify the new struct_hid_effect, but I am not sure there
is a point to prevent the haptic device of having that capability.
>
> If intensity modification for waveforms is supported by the device, the
> intensity control should be included in the manual trigger output report. This
> allows modification of the intensity on a per-waveform basis.
yep
So I think we should simply say that the haptic feedback needs to
follow HUTRR 63b, with the following amendments:
- continuous waveforms will be ignored by the host
- Repeat Count will be ignored by the host
- Auto mode trigger might get used for the autonomous mode described
above, but can be optional
>
> Alternatives Considered
> =======================
>
> Add a “true force” field to the multi-touch protocol
> ----------------------------------------------------
>
> This allows us to send “traditional pressure” in addition to “true force”. It
> also allows another possible protocol in addition to sending the force per
> contact:
>
> * Sending overall force and center of force: Should be easier to calculate
> depending on force sensor layout, and provides potentially useful extra info
>
> As mentioned before, there is already a concept of pressure in the multi-touch
> protocol, generally used as a proxy for contact area. We would add another field
> to represent force. However, the pressure field is defined by the protocol as
> actual pressure, and I feel this would only add more confusion. The touch_major
> and touch_minor fields can be used to report contact area explicitly, so we're
> not losing that capability.
Nope, I am not buying this idea. As mentioned above, the MT_PRESSURE
was always intended to report the pressure of the contact. It is an
approximation in the capacitive sensor case, but there is no point in
adding one new event. Also, if we set the INPUT_PROP_FORCEPAD, it is
now clear that the unit and thus the resolution is reliable.
>
> Report overall force, but not per-contact force
> -----------------------------------------------
>
> Instead of reporting the “traditional pressure,” send “true force” values in the
> ABS_PRESSURE field.The ABS_MT_PRESSURE field would still be used to send
> “traditional pressure” for each contact.
>
> I'm not convinced this is necessary, or the best idea. It makes backward
> compatibility easy, but other input libraries could detect forcepads by the
> resolution associated with the ABS_MT_PRESSURE field, and ignore that field,
> using touch_major and touch_minor instead.
>
> In addition, it adds even more confusion than the above option, so I think I'd
> prefer a clean break.
I think I like having the ABS_PRESSURE as the total force applied. But
we need per-contact force if this is available.
And again, I don't think we should mention the "traditional pressure"
as this is just a quirk from the capacitive sensors, and the actual
pressure was reported before those sensors came mainstream (think of
resistive ones).
>
> Use driver-wide gain for force feedback
> ---------------------------------------
>
> The force feedback protocol also has a mechanism to set driver-wide gain, which
> could be used to set global effect strength level. However, allowing different
> strength levels for each predefined effect would allow more flexibility e.g.: if
> the user wants to have low strength for UI feedback effects and high strength
> for button click effects.
I also think the best would be to have per effect intensity.
Anyway, thanks a lot for putting the effort in this.
The last bit I wanted to add is that we should define properly the
haptic and touch protocol in a way that we can support non simple
haptic HID controllers. For example, I doubt the Apple forcepad
follows the HID spec, but if the driver can provide the same level of
quality we require to mark a device as INPUT_PROP_FORCEPAD, we don't
really care if the haptic interface is HID or not.
Cheers,
Benjamin
>
> [0]: https://www.kernel.org/doc/Documentation/input/multi-touch-protocol.txt
> [1]: https://www.kernel.org/doc/Documentation/input/ff.txt
> [2]: https://www.usb.org/sites/default/files/hutrr63b_-_haptics_page_redline_0.pdf
> [3]: Usage ID 0x30 of HID usage table 0x0D. See chapter 16:
> https://www.usb.org/sites/default/files/documents/hut1_12v2.pdf
> [4]: See section 6.2.2.7 of the HID specification:
> https://www.usb.org/sites/default/files/documents/hid1_11.pdf
