Hi Benjamin and Peter,
I am refreshing this old thread in order to clarify some things
discussed here :)
I specifically got lost when reading about virtual IDs for effects.
Asking more inline.
On Thu, Apr 18, 2019 at 7:00 AM Benjamin Tissoires
<benjamin.tissoires@xxxxxxxxxx> wrote:
>
> On Fri, Apr 12, 2019 at 8:32 PM Sean O'Brien <seobrien@xxxxxxxxxxxx> wrote:
> >
> > On Fri, Apr 12, 2019 at 12:42 AM Peter Hutterer <peter.hutterer@xxxxxxxxx> wrote:
> > >
> > > Ahoi!
> > >
> > > On Thu, Apr 11, 2019 at 11:24:30AM -0700, Sean O'Brien wrote:
> > > > Hi Benjamin and Peter,
> > > >
> > > > Thanks for your detailed replies. I'm glad you found it interesting :)
> > > >
> > > > On Wed, Apr 10, 2019 at 5:47 PM Peter Hutterer <peter.hutterer@xxxxxxxxx> wrote:
> > > > >
> > > > > Hey!
> > > > >
> > > > > I just have a few notes on top of what Benjamin said.
> > > > >
> > > > > On Wed, Apr 10, 2019 at 06:51:45PM +0200, Benjamin Tissoires wrote:
> > > > > > Hi Sean,
> > > > > >
> > > > > > On Wed, Apr 10, 2019 at 1:29 AM Sean O'Brien <seobrien@xxxxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I'm currently working on designing an interface for controlling "forcepads";
> > > > > > > that is, touchpads with force sensors and haptic actuators. Below is my
> > > > > > > proposal for the protocol at both the userspace and HID interfaces. I would
> > > > > > > appreciate any feedback you might have.
> > > > > >
> > > > > > Thanks a lot for starting this discussion. This is indeed an
> > > > > > interesting topic, and I had a lengthy chat with Peter this morning :)
> > > > > >
> > > > > > First, I'd like to get a small clarification.
> > > > > > In the proposal, you are making reference to both the HID side (so the
> > > > > > hardware protocol) and the kernel side (the client protocol).
> > > > > > I read it that you are trying to build two things:
> > > > > > - the HID design proposal, aimed at hardware makers
> > > > > > - the kernel behavior on how to use it
> > > > > >
> > > > > > If that is the case, I am glad that you can put some pressure to force
> > > > > > a sensible protocol to hardware makers :)
> > > > > >
> > > > > > Though, I think we should make a clear separation between the 2. We
> > > > > > need to think of it as a whole, but the document should clearly put
> > > > > > the boundaries IMO.
> > > > > >
> > > >
> > > > You are correct - there are two levels here and I'll try to make the separation
> > > > more clear in future versions.
> > > >
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Sean O'Brien
> > > > > > > Chromium OS Touch/Input Team
> > > > > > >
> > > > > > > Background
> > > > > > > ==========
> > > > > > >
> > > > > > > There are multiple independent projects to develop a touchpad with force sensors
> > > > > > > and haptic actuators, instead of a traditional button. These “forcepads” have
> > > > > > > several advantages and potential uses; they allow clicking across the entire
> > > > > > > touchpad surface, adjusting the force requirement for clicks, haptic feedback
> > > > > > > initiated by UI, etc.
> > > > > > >
> > > > > > > Objective
> > > > > > > =========
> > > > > > >
> > > > > > > Develop a standard protocol to allow userspace applications to communicate with
> > > > > > > forcepads, and minimize duplicated code and effort.
> > > > > > >
> > > > > > > Requirements:
> > > > > > > 1. Support UI-initiated haptic feedback.
> > > > > > > 2. Allow userspace to control when button press and button release haptic
> > > > > > > effects are triggered. (Useful when detecting a false click, changing force
> > > > > > > thresholds, or sending context-dependent effects)
> > > > > > > 3. Allow a backward-compatible mode where the forcepad produces button press and
> > > > > > > button release effects autonomously.
> > > > > > > 4. Reveal force sensor readings to userspace applications.
> > > > > > >
> > > > > > > Proposal
> > > > > > > ========
> > > > > > >
> > > > > > > I propose standardized forcepad support in the linux kernel at both the HID and
> > > > > > > userspace interface.
> > > > > > >
> > > > > > > Userspace Interface
> > > > > > > -------------------
> > > > > > >
> > > > > > > Multitouch
> > > > > > > ..........
> > > > > > >
> > > > > > > The linux kernel has a well defined protocol [0] for multitouch devices,
> > > > > > > including touchpads. The protocol defines a field called ABS_MT_PRESSURE as a
> > > > > > > measure of the pressure applied by a contact. Unfortunately, it is not used
> > > > > > > that way in practice- instead it is generally used as an approximate measure of
> > > > > > > contact area. I will distinguish these concepts by calling them “true force” and
> > > > > > > “traditional pressure”.
> > > > > >
> > > > > > Well, yes and now. It depends on the underlying technology.
> > > > > > For capacitive sensors, "traditional pressure" if reported through
> > > > > > ABS_MT_PRESSURE is the surface contact area. This is a rough
> > > > > > equivalent to a pressure, assuming we all have somehow flexible
> > > > > > fingers, and roughly the same shape. But for resistive touchscreens,
> > > > > > IIRC we have already access to the "true force".
> > > > > >
> > > > > > IIRC, when both ABS_MT_MAJOR and ABS_MT_MINOR are reported, libinput
> > > > > > ignores ABS_MT_PRESSURE. So in a sense, there is no need to add a new
> > > > > > ABS event, this one works quite well.
> > > > >
> > > > > just as an extra note here: the kernel doc for ABS_MT_PRESSURE says "The
> > > > > pressure, in arbitrary units, on the contact area", so that's what we do in
> > > > > libinput (and the old synaptics driver) at least. We have arbitrary
> > > > > thresholds but they are by device or device series. So *how* the pressure
> > > > > was generated by the hardware itself is already meaningless because all it
> > > > > gives us is some continuous data without useful reference points.
> > > > > IOW, using ABS_MT_PRESSURE will work just fine.
> > > > >
> > > > > And we can't just assume "if resolution is set, units are $foo" because
> > > > > nothing written in the last decade or so will assume that. Some extra flag
> > > > > is needed, like INPUT_PROP_FORCEPAD.
> > > > >
> > > >
> > > > Great, I won't worry about the distinction then, except to say forcepads must
> > > > always use force and define units.
> > > >
> > > > > > >
> > > > > > > Instead of using ABS_MT_PRESSURE to report the “traditional pressure,” forcepads
> > > > > > > would send “true force” values in this field. For each contact, this field would
> > > > > > > report the estimated force applied by that contact.
> > > > > > >
> > > > > > > The resolution of ABS_MT_PRESSURE should also be defined and reported, so that
> > > > > >
> > > > > > Please define here what units the protocol expects. And please make it
> > > > > > international (g/m2, N/m2 or whatever). Though Newtons might be
> > > > > > difficult to grasp for people.
> > > >
> > > > I was actually thinking Newtons (or grams), instead of g/m2 or N/m2. The sensors
> > > > are more directly measuring force rather than pressure. The value could be
> > > > converted to pressure based on the contact area, but this raises the question of
> > > > how to report when force is applied to the pressure with no fingers, e.g.
> > > > pressing with an eraser or some object that doesn't trigger the capacitive
> > > > sensors. If we us N or g, this value can be reported in ABS_PRESSURE directly.
> > > > Otherwise I'm not sure how to convert to pressure units.
> > >
> > > I think N are just fine here. The evdev API is what it is and given the
> > > circumstances I don't think don't need to attempt to be scientifically exact
> > > here.
> > >
> > > > > > > userspace consumers can translate to force units. By defining the resolution, we
> > > > > > > also differentiate it from how it is used to report “traditional pressure”,
> > > > > > > where it has no resolution. Userspace consumers will be able to use this to
> > > > > > > detect that this is a forcepad, and treat the pressure field accordingly.
> > > > > >
> > > > > > I tend to disagree. It is best in that case to explicitly mark the
> > > > > > forcepad as such by adding a new INPUT_PROP. INPUT_PROP_FORCEPAD would
> > > > > > be nice.
> > > > > >
> > > > > > The reason is you don't know for sure that the reported unit is
> > > > > > correct for the existing devices. If you want this to actually matter,
> > > > > > then you can enforce the unit to be correct for a certified device,
> > > > > > and then only export INPUT_PROP_FORCEPAD for those devices (thinking
> > > > > > at the Win8 certification blob for multitouch screens/touchpads).
> > > > > >
> > > > > > Note that defining what an INPUT_PROP_FORCEPAD is would be nice. For
> > > > > > example does the user controlled haptic feedback be a requirement.
> > > > > >
> > > > > > I would suggested:
> > > > > > - can differentiate between at least 5 fingers
> > > > > > - correct resolution for the X/Y (units and value)
> > > > > > - report correct force per touch, and correct units for them
> > > > > > - follows the MT protocol type B
> > > > > >
> > > > > > So for hardware vendors, we would require to follow the MS spec for
> > > > > > input devices in Win8 and Win8.1, and in addition support the Simple
> > > > > > haptic controller HID table, and report correct units for the
> > > > > > pressure.
> > > > > >
> > > >
> > > > Agreed, adding a new INPUT_PROP is much cleaner, and acts as a convenient label
> > > > for describing where this protocol should be followed.
> > > >
> > > > > > >
> > > > > > > ABS_PRESSURE may be optionally reported as the total force applied to the
> > > > > > > forcepad.
> > > > > > >
> > > > > > > The device/driver shouldn’t detect button clicks, this is left to the userspace
> > > > > > > gesture library. Accordingly, the driver should not sent BTN_* events to
> > > > > > > userspace in normal operating mode. However it should still report the ability
> > > > > > > to produce such events, for use in autonomous mode.
> > > > > >
> > > > > > For backward compatibility, and to be able to debug it properly, you
> > > > > > should keep the BTN_* events emulated in all cases.
> > > > > > The userspace can ignore the events it doesn't want this way, but you
> > > > > > will be able to debug the btn emulations on your current session
> > > > > > without having to kill your compositor.
> > > > > > There shouldn't be much of a head over forwarding those events, as it
> > > > > > will never come alone, and will always be with an other one at least
> > > > > > (pressure being 0 or less).
> > > > > >
> > > > > > Also, not sending BTN_TOUCH and BTN_LEFT might give some headaches to
> > > > > > legacy applications.
> > > > > >
> > > >
> > > > This also makes the difference between host-controlled and autonomous mode
> > > > smaller, so this sounds good to me.
> > > >
> > > > > > >
> > > > > > > Haptic Control
> > > > > > > ..............
> > > > > > >
> > > > > > > The force feedback protocol [1] should be used to request predefined effects.
> > > > > >
> > > > > > s/request/control/
> > > > > >
> > > > > > >
> > > > > > > Typical use of the force feedback protocol requires loading effects to the
> > > > > > > driver by describing the output waveform, and then requesting those effects
> > > > > > > using an ID provided by the driver. We don’t want to describe the output
> > > > > > > waveform explicitly, but instead use a set of predefined IDs for the desired
> > > > > > > effect types. The device/driver would be responsible for having the effects
> > > > > > > loaded and ready to respond to a request for the predefined IDs.
> > > > > >
> > > > > > Re-reading through this made my head a little bit clearer.
> > > > > >
> > > > > > I think I am starting to see what you are saying, but the proposal
> > > > > > would need to be more precised here.
> > > > > >
> > > > > > >
> > > > > > > The force feedback protocol will need to be extended to allow requests for
> > > > > > > predefined IDs. This requires a new feedback effect type:
> > > > > > >
> > > > > > > /**
> > > > > > > * struct ff_predefined_effect
> > > > > > > * @level: strength of the effect
> > > > > > > * @vendor: ID of the vendor who defined the effect
> > > > > > > * @id: ID of the effect among those defined by the vendor
> > > > > > > */
> > > > > > > struct ff_predefined_effect {
> > > > > > > __s16 level;
> > > > > > > __u16 vendor;
> > > > > > > __u16 id;
> > > > > > > }
> > > > > > >
> > > > > > > Vendors can define specifications for the waveforms and assign them IDs. They
> > > > > > > could then be requested using their vendor ID and the waveform ID, as defined in
> > > > > > > the simple haptic HID protocol [2].
> > > > > > >
> > > > > > > To allow a standard way to trigger press and release effects, all forcepads
> > > > > > > should support the WAVEFORM_PRESS and WAVEFORM_RELEASE through this interface.
> > > > > > > Since the standard waveform id namespace doesn’t overlap with the vendor
> > > > > > > waveform id namespace, the vendor id can be ignored for these waveforms.
> > > > > >
> > > > > > First of all, I think you should split the HID (device/firmware)
> > > > > > requirements from how they are used in the kernel.
> > > > > >
> > > > > > IMO, it's fine to say that only touchpads exporting the "simple haptic
> > > > > > HID protocol [2]" will be supported by this proposal.
> > > > > >
> > > >
> > > > This is position I'd like to take, to discourage a proliferation of different
> > > > protocols.
> > > >
> > > > > > And FWIW, the Microsoft Surface Dial already exposes this HID
> > > > > > collection, and can be used to toy around the proposal (just saying).
> > > > > >
> > > > > > If you intend to be more generic, we should not refer to this HID
> > > > > > collection but explain how things are going to be in shape in the
> > > > > > kernel.
> > > > > >
> > > > > > That being said, there are a few additions/changes I'd like to see.
> > > > > >
> > > > > > The way the input_ff protocol works is:
> > > > > > - userspace upload an effect through EVIOCSFF and stores it in an id
> > > > > > - this effect is supposedly stored into the device itself
> > > > > > - userspace can then play the effect whenever it wants by calling the
> > > > > > id in an input_event written to the evdev node
> > > > > > - eventually, userspace can remove the effect with EVIOCRMFF
> > > > > >
> > > > > > This doesn't entirely matches the simple haptic HID protocol.
> > > > > >
> > > > > > As I read the documentation, the simple haptic HID only defines a set
> > > > > > of waveform types:
> > > > > > WAVEFORM_NONE, WAVEFORM_STOP, WAVEFORM_CLICK,
> > > > > > WAVEFORM_BUZZ_CONTINUOUS, WAVEFORM_RUMBLE_CONTINUOUS, WAVEFORM_PRESS,
> > > > > > and WAVEFORM_RELEASE. It also defines a "vendor" range where vendors
> > > > > > can put any waveform they see fit and define in their haptic device.
> > > > > >
> > > > > > So, I think that's why you want to augment the ff protocol by adding
> > > > > > the struct ff_predefined_effect.
> > > > > >
> > > > > > After thinking more about that, you are correct, we can not express
> > > > > > those "simple haptic HID" waveforms in the current ff protocol. But I
> > > > > > think the new struct should be focused on HID, not vendors, and
> > > > > > contains the whole effect settings. All the current struct ff_*_effect
> > > > > > also have the period, the magnitude, the offset, etc...
> > > > > >
> > > > > > Re-reading through the HID HUTRR, I think we should add Intensity,
> > > > > > Repeat Count, and Duration.
> > > > > >
> > > > > > So I would prefer to have a more generic ff_effect:
> > > > > > struct ff_hid_effect {
> > > > > > __u16 hid_usage;
> > > > > > __s16 intensity;
> > > > > > __u16 repeat_count;
> > > > > > __u16 duration;
> > > > > > }
> > > > > >
> > > > > > The id is already part of struct ff_effect, so this should allow us to
> > > > > > "upload" a predefined hid effect.
> > > > > >
> > > > > > Note that as I read it, we do not want auto-triggered haptic feedback
> > > > > > here. If we do, we would nee to add an other field to the
> > > > > > ff_hid_effect struct.
> > > > > >
> > > > > >
> > > > > > The next thing I'd like to get some clarifications is how these
> > > > > > effects are used. As I read it, userspace "uploads" the effect, and
> > > > > > uses it, but the HID HUTRR doesn't say that the effects and parameters
> > > > > > are stored in the device itself. It is for the auto triggered ones,
> > > > > > but given that we want to opt for the manual trigger, we should store
> > > > > > it ourselves.
> > > > > >
> > > >
> > > > My understanding is that all the simple haptic effects are indeed stored in the
> > > > device. Certain parameters (intensity, repeat_count, retrigger_period) are
> > > > dynamic, but the list of available waveforms (WAVEFORM_PRESS etc.) and their
> > > > associated durations are static properties of the device.
> > > >
> > > > In fact, I think we will want to add a mechanism for userspace to ask for a list
> > > > of all available waveforms, instead of querying for each specific waveform it
> > > > may be interested in.
> > >
> > > Do we have a specific use-case for this? APIs that allow listing things
> > > that are otherwise basically random are mostly useful for debugging tools
> > > (where performance isn't usually an issue) but nothing else. If you are a
> > > userspace process and I give you a list of WAVEFORM_PRESS, WAVEFORM_RELEASE,
> > > 0x1231113, 0x11aaab1123 and 0x112abff11, what will you do with that
> > > information? if you have #defines for some of them (e.g. press, release) you
> > > can query them directly. If not - does it matter what else I support?
> > >
> > > As a library author, I can definitely say that an API that only allows "do
> > > you have capability $FOO" is a lot simpler to support than "give me all
> > > capabilities you know of".
> > >
> >
> > Makes sense, that works for me.
>
> [coming back after 4 days of PTO]
>
> The one thing I wanted to add here is that I am not so sure we need an
> API to know what capabilities are present in the device:
> - they are exported in the report descriptor, so a call to fetch it
> from hidraw or the sysfs is theoretically enough to get the
> information (there is the parsing, but... heh)
> - they have a one time usage, when you connect the device (at boot
> likely) and according to the discussion should be set by an external
> helper.
>
> So I would suggest to keep things to the minimum (i.e. not having a
> query API), get a first design proposal in addition to the spec
> proposal and then decide on whether we need a brand new kernel API for
> that.
>
> Also, we could skip this step entirely by having a helper that tries
> to store 0x1231113 in id 5 and expect an error if the feature is not
> available.
> It is not that bad if we have a userspace db of devices in the end.
>
> Cheers,
> Benjamin
>
> >
> > > > So we would add FF_SIMPLE to the features bitfield. If this feature is
> > > > supported, userspace can use a new ioctl to find the supported simple effects:
> > > >
> > > > struct input_ff_request_simple {
> > > > __u16 vendor_id;
> > > > __u16 vendor_page;
> > > > __u16 hid_usages[num_effects];
> > > > __u16 durations[num_effects];
> > > > }
> > > >
> > > > In this case vendor_id and vendor_page apply to all the waveforms in the vendor
> > > > range.
> > > >
> > > > Then userspace can "upload" effects with:
> > > >
> > > > struct ff_hid_effect {
> > > > __u16 hid_usage;
> > > > __s16 intensity;
> > > > __u16 repeat_count;
> > > > __u16 retrigger_period;
> > > > }
> > > >
> > > > > > So, the solution we came to this morning, while talking to Peter, was
> > > > > > that the HID driver for a simple haptic HID device would allocate a
> > > > > > virtual device memory to store the effects and the parameters.
> > > > > >
> > > > > > This way, we can:
> > > > > > - upload effect WAVEFORM_RELEASE with its parameters in id 0 of the
> > > > > > drvdata of the device
> > > > > > - upload effect WAVEFORM_PRESS with its parameters in id 1 of the
> > > > > > drvdata of the device
> > > > > > - ...
> > > > > > - upload effect WAVEFORM_VENDOR_ZZZ_ZZZ with its parameters in id N of
> > > > > > the drvdata of the device -> userspace will use it while scrolling for
> > > > > > instance
> > > > > > - ...
> > > > > >
> > > > > > Then the kernel on BTN_LEFT press can automatically trigger the effect
> > > > > > with id 1 and the one with id 0 on release in the case of the
> > > > > > autonomous mode mentioned below.
> > > > > >
> > > > > > To solve the question of knowing which effect should be loaded in
> > > > > > which slot, I think we should rely on a userspace helper (udev?).
> > > > > > We definitively not want the kernel to keep a list of devices to
> > > > > > effects matches, but having a udev database (hwdb and intrinsic?)
> > > > > > would nicely solve the issue as we do not need to update the kernel
> > > > > > for each new device coming in.
> > > > > >
> > > > > > From the kernel driver, we can populate the WAVEFORM_PRESS and
> > > > > > WAVEFORM_RELEASE with some sensible parameters, but userspace should
> > > > > > be allowed to override them.
> > > > > >
> > > > > > The advantage of having this virtual memory of device effects, is that
> > > > > > each userspace implementation could use its own matching for effects.
> > > > > > For example, libinput might want to say:
> > > > > > - id 0 -> BTN_LEFT released
> > > > > > - id 1 -> BTN_LEFT pressed
> > > > > > - id 0x1000 -> scrolling up
> > > > > > - id 0x1001 -> scrolling down
> > > > > > - id 0x2042 -> hard press
> > > > > >
Was there some idea up then to implement virtual effect IDs? Right now
it seems that
the number of possible FF effects is limited to FF_MAX_EFFECTS and
that it is the kernel
and not the user space that assigns the ID when an effect is uploaded.
Or was it more of a suggestion for the future than a requirement
regarding the simple haptic
feedback implementation?
> > > > > > But chromeOS might use different ids and different meaning for them.
> > > > > > The first 2 ids first should be reserved for the kernel button left
> > > > > > emulation though.
> > > > > >
> > > > > > Note that given that the effect memory is 'virtual' in the driver, we
> > > > > > can use any id and we are not limited to a few numbers of them.
> > > > > >
> > > > > > An other thing is that we do not need to require WAVEFORM_PRESS and
> > > > > > WAVEFORM_RELEASE. If they are set in the haptic device, the kernel can
> > > > > > set them for us, but if they are not, userspace can always decide to
> > > > > > put one of the other waveforms in ids 0 and 1.
> > > > > >
> > > > > > We should mention that not defining these two waveforms will induce a
> > > > > > state were no haptic feedback will be automatically loaded from the
> > > > > > kernel though (userspace will have to do it).
> > > >
> > > > True, I guess I would say it's strongly encouraged to include WAVEFORM_PRESS and
> > > > WAVEFORM_RELEASE for that reason.
> > > >
> > > > > >
> > > > > > >
> > > > > > > Autonomous mode
> > > > > > > ---------------
> > > > > > >
> > > > > > > In order to facilitate an operating system which cannot handle force feedback,
> > > > > > > the forcepad should start up in “autonomous mode”, meaning it acts as a normal
> > > > > > > touchpad. This means it should perform the press and release haptic feedback
> > > > > > > autonomously at predefined force thresholds, and send the appropriate BTN_*
> > > > > > > events.
> > > > > >
> > > > > > I think that if we were to follow my idea of using effect ids 0 and 1
> > > > > > for release/press, we should mention that the kernel will play the
> > > > > > effects stored as these ids.
> > > > > >
> > > > > > >
> > > > > > > After verifying that all of the required haptic effects are available through
> > > > > > > the force feedback protocol, the OS can enable host-controlled mode. This could
> > > > > > > be done by writing to a sysfs node “host_feedback_enabled”.
> > > > > >
> > > > > > You don't really want a sysfs file here. Because if the input part
> > > > > > generating the haptic feedback crashes, or if you are VT switching and
> > > > > > your evdev node gets revoked, you are left in a system without haptic
> > > > > > feedback for clicks.
> > > > >
> > > > > Extra note here: compositors don't have write access to sysfs. some do, but
> > > > > the default assumption is that they don't.
> > > > >
> > > > > > So, our idea was to use a new couple of ioctls, EVIOCFFTAKEMASTER and
> > > > > > EVIOCFFRELEASEMASTER (or whatever is more politically admitted), which
> > > > > > the kernel would ref count on each effect id.
> > > > > >
> > > > > > So, if the OS decides to take control over the press autonomous haptic
> > > > > > feedback, it will emit the ioctl EVIOCFFTAKEMASTER with the effect id
> > > > > > 1. If an other client also wants to control this id, it can also take
> > > > > > the control, and the total count of clients controlling this effect id
> > > > > > is now 2. If a BTN_LEFT is emitted by the device (or emulated by the
> > > > > > kernel), the driver will check the refcount of the effect id, sees
> > > > > > that there is at least one, and will not forward the haptic effect to
> > > > > > the device.
> > > > > > On the contrary, in this case, effect id 0 is not handled by
> > > > > > userspace, and the kernel will control it when a BTN_LEFT release is
> > > > > > emitted.
> > > > > >
> > > > > > Note that this ioctl will only have an impact on effect ids 0 and 1.
> > > > >
> > > > > hmm, I don't think there's a technical reason for that limitation, is there?
> > > > > simply specifying it as "any effect ID that does not have a current master
> > > > > will not get auto-played by the kernel". Simple enough and leaves the actual
> > > > > usage more flexible.
> > > > >
> > > > > > The benefit of using an ioctl is that is the client closes the fd, or
> > > > > > if the fd gets revoked, the kernel can decrement de usage count and
> > > > > > eventually switch back to the autonomous mode described above.
> > > > >
> > > > > I'll add a shorter summary for the normal case, just to make it easier to
> > > > > comprehend the whole thing:
> > > > > - the kernel has default effects for ids 0/1
> > > > > - some process (udev) uploads device-specific waveforms where needed, for ids
> > > > > 0/1 and others if need be
> > > > > - the kernel always sends BTN_LEFT when some hardcoded threshold is met
> > > > > - when sending BTN_LEFT, the kernel plays id 0/1 for release/press
> > > > > - if a process has the EVIOCFFTAKEMASTER for a given id, the kernel does not
> > > > > play it and relies on that client to do so.
> > > > >
> > > > > This approach means:
> > > > > - default behaviour is predictable and useful
> > > > > - updating for devices is in userspace
> > > > > - the entity controlling the click-buzz (e.g. libinput) doesn't need to be
> > > > > aware of the device-specific waveforms
> > > > > - VT-switch/compositor change/suspend/... (i.e. close(fd)) restores default
> > > > > behaviour
> > > > > - other processes can use the pad for haptic feedback without interfering or
> > > > > having to care about whether the session supports forcepads for button
> > > > > handling
> > > > >
> > > > > > > When the host enters suspend mode, the OS will not be able to respond quickly
> > > > > > > enough to input from the touchpad to tell it to perform haptic feedback, making
> > > > > > > the touchpad feel unresponsive. When the host suspends, the touchpad should
> > > > > > > enter autonomous mode.
> > > > > >
> > > > > > This could be achieved in my proposal above by either closing the fd,
> > > > > > or revoking it, so we would get this for free :)
> > > > > >
> > > > > > >
> > > > > > > HID Interface
> > > > > > > -------------
> > > > > >
> > > > > > So this is the part where we tell ODM what to do. Correct?
> > > > > >
> > > > > > >
> > > > > > > Multitouch
> > > > > > > ..........
> > > > > > >
> > > > > > > The HID API for multitouch reports is mostly unchanged except:
> > > > > > >
> > > > > > > 1. The tip pressure field [3] should be used to report “true force” instead of
> > > > > > > “traditional pressure”. The physical unit type, exponent, and limits should
> > > > > > > be reported in the report descriptor for the “true force” field [4].
> > > > > >
> > > > > > works for me
> > > > > >
> > > > > > > 2. The device will always report it’s button as being unpressed, except in
> > > > > > > autonomous mode, when it will report the button state according to its
> > > > > > > predefined force thresholds.
> > > > >
> > > > > fwiw, devices that report capabilities but never send those events are a
> > > > > nightmare to deal with because it forces heuristics and guesswork in the
> > > > > clients that can be wrong at any point. Filtering events because we pretend
> > > > > to know better than the kernel driver is a lot easier.
> > > > >
> > > > > Cheers,
> > > > > Peter
> > > > >
> > > > >
> > > > > > I think I would prefer having a basic left button emulation that we
> > > > > > can rely on when userspace doesn't know about forcepads.
> > > > > > This would allow us to not have to emulate it in the kernel, but just
> > > > > > forward the data as it comes in.
> > > > > >
> > > > > > Also, this would allow the driver to not have to drive the haptic
> > > > > > feedback in the autonomous mode: we just set the auto trigger on
> > > > > > BTN_LEFT usage, and done.
> > > > > > (if we do not enforce the auto triggering, then the driver will also
> > > > > > have to manually send the haptic events on BTN_LEFT 0/1).
> > > > > >
> > > >
> > > > This all sounds good to me. However, I don't think I was entirely clear. When I
> > > > said "autonomous mode", I meant more "device-controlled". So really there are
> > > > three modes to think about:
> > > >
> > > > - Userspace-controlled: Some process has emited ioct EVIOCFFTAKEMASTER for ids
> > > > 0/1. The device will emit BTN_LEFT at some hardcoded force threshold. The
> > > > kernel will forward those events to userspace but will not play haptic
> > > > effects.
> > > > - Kernel-controlled: Nobody has taken control of ids 0/1 so the kernel will
> > > > default to playing id 0/1 whenever BTN_LEFT changes.
> > > > - Device-controlled: The auto trigger is set on the device, associated with
> > > > BTN_LEFT.
> > > >
> > > > I want device-controlled mode because if the system suspends, I want the device
> > > > to be able to respond with haptic effects for press and release quickly. So the
> > > > kernel should set the auto trigger when suspending.
> > > >
> > > > However, there is a limitation of the HUTRR: only one waveform can be set as the
> > > > auto trigger, so we would have to choose one waveform to be played for both
> > > > press and release. Therefore, I would like to use kernel-controlled mode when
> > > > the system is not suspended, so it can send the correct waveform.
> > >
> > > We're blurring the HID vs evdev distinction and I'm not sure which one
> > > you're discussion here but: from userspace, kernel vs device-controlled is
> > > an implementation detail of the kernel. so is having only one auto-trigger,
> > > just play the same one for press release when you must, and do the right
> > > thing otherwise.
> > >
> > > probably obvious and we all agree here, but let's state it out loud at least
> > > once :)
> > >
> >
> > That's exactly right. Userspace won't know the difference between kernel or
> > device-controlled. And the device won't know the difference between userspace
> > and kernel-controlled.
> >
> > Best Regards,
> > Sean
> >
> > > Cheers,
> > > Peter
> > >
> > >
> > > > > > If we are giving requirements to ODM, I think we also need:
> > > > > >
> > > > > > 3. a specific application usage that tells us that the device is a forcepad.
> > > > > > Ideally, I would love to extend the HUT, but we might consider saying
> > > > > > that a forcepad, to be exposed as such, needs to expose a "Simple
> > > > > > Haptic Controller" logical collection alongside the Touch Pad one.
> > > > > >
> > > > > > That's what the Surface dial is doing:
> > > > > > - application collection: System Multi-Axis Controller
> > > > > > - logical collection: Puck
> > > > > > - logical collection: Simple Haptic Controller
> > > > > > - logical collection: Durations List
> > > > > > - logical collection: Waveforms List
> > > > > > - endof application collection
> > > > > >
> > > > > > So this would make clear what a HID forcepad is.
> > > >
> > > > Sounds good.
> > > >
> > > > > >
> > > > > > 4. a way to store in the HID device the pressure threshold the device
> > > > > > is supposed to report the click.
> > > > > > We can guess which threshold it is if the unit is correct, but I think
> > > > > > I would rather have the device telling us this. Problem is, I can not
> > > > > > find a good HID usage for that.
> > > >
> > > > Ah yes, I had thought about that a little too. Like you, I couldn't find a good
> > > > HID usage. I'd appreciate any alternate suggestions for that.
> > > >
> > > > > >
> > > > > > 5. If a device doesn't meet those criteria above, it will not get
> > > > > > categorized as a FORCEPAD and will not get all the fancy FF effects
> > > > > > from the OS.
> > > > > >
> > > > > > This last point is very important as this will prevent us to have to
> > > > > > deal with quirky devices.
> > > > > >
> > > > > > Note that MS solved the issue by requiring ODM to actually test their
> > > > > > devices, and then MS issues a blob in a vendor feature that is tested
> > > > > > to enable or not the touchscreen / touchpad gestures.
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Haptic control
> > > > > > > ..............
> > > > > > >
> > > > > > > The simple haptic HID protocol [2] should be used.
> > > > > >
> > > > > > s/should/must/
> > > > > >
> > > > > > >
> > > > > > > The following waveforms should be supported:
> > > > > > >
> > > > > > > | WAVEFORM_NONE | Implicit waveforms required by protocol |
> > > > > > > | WAVEFORM_STOP | |
> > > > > > > | ------------------------ | ------------------------------------------------- |
> > > > > > > | WAVEFORM_PRESS | To be used in autonomous mode and host-controlled |
> > > > > > > | WAVEFORM_RELEASE | mode to simulate button press and release. |
> > > > > >
> > > > > > I would not absolutely require those 2. If they are present, the
> > > > > > kernel (driver) should set them as the default feedback (in the
> > > > > > autonomous mode), but if they aren't, it's not a big deal, the
> > > > > > userspace will have to quirk the device.
> > > > > >
> > > > > > > | ------------------------ | ------------------------------------------------- |
> > > > > > > | Vendor-defined waveforms | Optional waveforms to be used in host-controlled |
> > > > > > > | | mode, subject to vendor specification. |
> > > > > >
> > > > > > I think we should just say that the device is free to implement all
> > > > > > the waveforms it wants as per HUTRR 63b.
> > > >
> > > > Fair enough, as long as at least one waveform is declared (as is required by
> > > > HUTRR 63b) it can be designated as a forcepad. WAVEFORM_PRESS and
> > > > WAVEFORM_RELEASE are recommended but not required.
> > > >
> > > > > >
> > > > > > >
> > > > > > > All waveforms will have an associated duration; continuous waveforms will not be
> > > > > > > supported.
> > > > > > >
> > > > > > > Only manual triggering will be supported through this interface. Autonomous
> > > > > > > triggering of waveforms is enabled by putting the device in autonomous mode.
> > > > > >
> > > > > > This part is ambiguous. The Auto triggering on the HUTRR 63b says that
> > > > > > when the device emits a specific HID usage, it will also emit the
> > > > > > matching haptic effect.
> > > > > > While here, you want the kernel to driver the button press/release
> > > > > > (unless I am reading it wrong).
> > > > > >
> > > > > > Also, I think you are mixing here the HID requirements and the
> > > > > > exported kernel API. I think there is nothing wrong to comply with the
> > > > > > HUTRR 63b. We can use the auto-trigger functionality in the autonomous
> > > > > > mode described above. The point is that we do not want to export that
> > > > > > to the userspace.
> > > > > >
> > > >
> > > > Yes, you are right. We can achieve the various autonomous modes
> > > > (kernel-controlled and device-controlled) through HUTRR 63b using manual and
> > > > auto triggering respectively.
> > > >
> > > > > > >
> > > > > > > Retriggering (queueing multiple triggers of the same waveform) is not supported.
> > > > > >
> > > > > > This will simplify the new struct_hid_effect, but I am not sure there
> > > > > > is a point to prevent the haptic device of having that capability.
> > > > > >
> > > > > > >
> > > > > > > If intensity modification for waveforms is supported by the device, the
> > > > > > > intensity control should be included in the manual trigger output report. This
> > > > > > > allows modification of the intensity on a per-waveform basis.
> > > > > >
> > > > > > yep
> > > > > >
> > > > > > So I think we should simply say that the haptic feedback needs to
> > > > > > follow HUTRR 63b, with the following amendments:
> > > > > > - continuous waveforms will be ignored by the host
> > > > > > - Repeat Count will be ignored by the host
> > > > > > - Auto mode trigger might get used for the autonomous mode described
> > > > > > above, but can be optional
> > > > > >
> > > >
> > > > I've been convinced that there's no harm in supporting retriggering, and that in
> > > > fact we should encourage the use of auto trigger so the device can perfom
> > > > effects when the system is suspended.
> > > >
> > > > Best regards,
> > > > Sean
> > > >
> > > > > > >
> > > > > > > Alternatives Considered
> > > > > > > =======================
> > > > > > >
> > > > > > > Add a “true force” field to the multi-touch protocol
> > > > > > > ----------------------------------------------------
> > > > > > >
> > > > > > > This allows us to send “traditional pressure” in addition to “true force”. It
> > > > > > > also allows another possible protocol in addition to sending the force per
> > > > > > > contact:
> > > > > > >
> > > > > > > * Sending overall force and center of force: Should be easier to calculate
> > > > > > > depending on force sensor layout, and provides potentially useful extra info
> > > > > > >
> > > > > > > As mentioned before, there is already a concept of pressure in the multi-touch
> > > > > > > protocol, generally used as a proxy for contact area. We would add another field
> > > > > > > to represent force. However, the pressure field is defined by the protocol as
> > > > > > > actual pressure, and I feel this would only add more confusion. The touch_major
> > > > > > > and touch_minor fields can be used to report contact area explicitly, so we're
> > > > > > > not losing that capability.
> > > > > >
> > > > > > Nope, I am not buying this idea. As mentioned above, the MT_PRESSURE
> > > > > > was always intended to report the pressure of the contact. It is an
> > > > > > approximation in the capacitive sensor case, but there is no point in
> > > > > > adding one new event. Also, if we set the INPUT_PROP_FORCEPAD, it is
> > > > > > now clear that the unit and thus the resolution is reliable.
> > > > > >
> > > > > > >
> > > > > > > Report overall force, but not per-contact force
> > > > > > > -----------------------------------------------
> > > > > > >
> > > > > > > Instead of reporting the “traditional pressure,” send “true force” values in the
> > > > > > > ABS_PRESSURE field.The ABS_MT_PRESSURE field would still be used to send
> > > > > > > “traditional pressure” for each contact.
> > > > > > >
> > > > > > > I'm not convinced this is necessary, or the best idea. It makes backward
> > > > > > > compatibility easy, but other input libraries could detect forcepads by the
> > > > > > > resolution associated with the ABS_MT_PRESSURE field, and ignore that field,
> > > > > > > using touch_major and touch_minor instead.
> > > > > > >
> > > > > > > In addition, it adds even more confusion than the above option, so I think I'd
> > > > > > > prefer a clean break.
> > > > > >
> > > > > > I think I like having the ABS_PRESSURE as the total force applied. But
> > > > > > we need per-contact force if this is available.
> > > > > > And again, I don't think we should mention the "traditional pressure"
> > > > > > as this is just a quirk from the capacitive sensors, and the actual
> > > > > > pressure was reported before those sensors came mainstream (think of
> > > > > > resistive ones).
> > > > > >
> > > > > > >
> > > > > > > Use driver-wide gain for force feedback
> > > > > > > ---------------------------------------
> > > > > > >
> > > > > > > The force feedback protocol also has a mechanism to set driver-wide gain, which
> > > > > > > could be used to set global effect strength level. However, allowing different
> > > > > > > strength levels for each predefined effect would allow more flexibility e.g.: if
> > > > > > > the user wants to have low strength for UI feedback effects and high strength
> > > > > > > for button click effects.
> > > > > >
> > > > > > I also think the best would be to have per effect intensity.
> > > > > >
> > > > > > Anyway, thanks a lot for putting the effort in this.
> > > > > >
> > > > > > The last bit I wanted to add is that we should define properly the
> > > > > > haptic and touch protocol in a way that we can support non simple
> > > > > > haptic HID controllers. For example, I doubt the Apple forcepad
> > > > > > follows the HID spec, but if the driver can provide the same level of
> > > > > > quality we require to mark a device as INPUT_PROP_FORCEPAD, we don't
> > > > > > really care if the haptic interface is HID or not.
> > > > > >
> > > > > > Cheers,
> > > > > > Benjamin
> > > > > >
> > > > > > >
Regards,
Angela
> > > > > > > [0]: https://www.kernel.org/doc/Documentation/input/multi-touch-protocol.txt
> > > > > > > [1]: https://www.kernel.org/doc/Documentation/input/ff.txt
> > > > > > > [2]: https://www.usb.org/sites/default/files/hutrr63b_-_haptics_page_redline_0.pdf
> > > > > > > [3]: Usage ID 0x30 of HID usage table 0x0D. See chapter 16:
> > > > > > > https://www.usb.org/sites/default/files/documents/hut1_12v2.pdf
> > > > > > > [4]: See section 6.2.2.7 of the HID specification:
> > > > > > > https://www.usb.org/sites/default/files/documents/hid1_11.pdf
