Richer Touch events / unified pointer event model

[chris-morgan]

I note Touch, but it’s rather limited. My Surface Book, for example, has a pressure-sensitive pen, which neither Touch nor Mouse can model; and the latest such pens support tilt as well.

I suspect it’s worth looking into something like the Pointer Events API which browsers have all comparatively recently taken up.

[marcianx]

I'd also like to note rust-windowing/glutin#916 which I was asked to forward to here:

The Touch event does not include a touch radius or any other obvious way to distinguish finger and stylus touch events. This is particularly useful for palm rejection on large screens when using a stylus. Could such support be added?

[tangmi]

Would a precursor to this be getting pen/tablet support first (#99)?

Edit: some literature that might help implementing pointer events cross-platform: https://gist.github.com/k3a/ef98593a3571b1b399169448327ad333

Edit2: It seems like there needs to be a decision on whether to keep Mouse/Touch/Pen separate or unify them under a "PointerEvent" type. Perhaps winit could redundantly fire the separate events with the unified and let the app decide whether to handle them separately or unified (like how the web pointer events api currently is?)

[Osspial]

@tangmi I think it goes the other way - getting richer touch support will lay the ground for adding pen/tablet events. We've slowly been making progress towards richer touch events, via recent touch pressure PRs for iOS and Windows, and I expect that once more cross-platform support for rich touch events has been added pen/tablet support won't be too much of a technical or API jump.

It seems like there needs to be a decision on whether to keep Mouse/Touch/Pen separate or unify them under a "PointerEvent" type.

That's... an intriguing idea, actually. I'd support unifying pointer events into a single type, especially since that'll improve usability in the naive case where an application is designed for just one input type and given input of a format it didn't necessarily expect.

[chris-morgan]

The Pointer Events API which I suggested as a model is all about unified event types. If you’re not familiar with it, please do read up on it before implementing anything in Winit.

The web still does click events for touch and pen, mousedown et al for pen events, and scrolling/zooming for touch events (adjustable via the touch-action CSS property). Naive users should use those things rather than pointerdown et al. This is a sane model for more than just the web’s backwards compatibility, because touch should generally act differently from a mouse or pen.

[tangmi]

@Osspial I wanted to try pen input on web through winit, so I've hacked together basic support for pointer events in winit-legacy (diff here: 2e11615...tangmi:winit-legacy-2).

Some observations that may already be known:

• A unified pointer event API was super easy to consume from my app's point of view, although there needs to be some thought done on e.g. what pointerdown (and no button pressed) vs mousedown means, scrolling events, etc. But, I think there definitely is an argument for more semantic events on top (like mouse click, double click, two-finger zoom/pan, etc), maybe as a library that can hook into the custom events on winit 0.20?
• The Windows provides APIs to handle all inputs in a given frame at once (which allows for handling e.g. multitouch gestures). winit (legacy) currently doesn't provide support for this?
• Switching from multiple input event kinds to a single input event kind was disruptive enough to the API that I decided to leave the mouse events as is to keep support in third-party libraries (like imgui-winit-support). I wonder if the pointer events/richer touch event support should be considered before 0.20 is released?

Notes on this changeset:

• Only implemented for web (via an ugly stdweb shim) and windows
• I've routed touch and pen events (but not mouse events) through pointer events (and disabled the touch event from winit)
• I don't fully understand how to call the pointerinfo APIs on windows, so on some hardware pen input skips?

Cheers!

[Osspial]

@tangmi Thank you so much for investigating this!

A unified pointer event API was super easy to consume from my app's point of view, although there needs to be some thought done on e.g. what pointerdown (and no button pressed) vs mousedown means, scrolling events, etc.

I think the cleanest way to expose input-specific events would be to have an enum field within Pointer that contains any non-shared data. I don't know what to do about other sorts of events (like scrolling), though - that'll warrant some further thought.

The Windows provides APIs to handle all inputs in a given frame at once (which allows for handling e.g. multitouch gestures). winit (legacy) currently doesn't provide support for this?

I don't fully understand how to call the pointerinfo APIs on windows, so on some hardware pen input skips?

Our touch handling has been improved quite a bit on master, actually (see #991 and #1134). I'm not familiar with the specifics of how the Windows touch API works, but I'd guess that those features would be easier to support with the new implementation. That implementation also seems to handle pointerinfo gracefully.

Switching from multiple input event kinds to a single input event kind was disruptive enough to the API that I decided to leave the mouse events as is to keep support in third-party libraries (like imgui-winit-support). I wonder if the pointer events/richer touch event support should be considered before 0.20 is released?

0.20 is already big enough - I'd rather not expand its scope further and end up in a perpetual alpha. I'm totally fine with removing the old-style mouse events in a future release, though. When we're improving our APIs, we should kill the clunky, thrown-together APIs while we aren't tied to stability guarantees.

[tangmi]

@Osspial: I'm revisiting this because I got some emails on other issues relating to this... Since 0.20 got released, is there any additional blockers preventing this? I think I can port my Windows impl to head and start the investigation/discussion for other platforms?

[kirawi]

@tangmi I was investigating doing that myself before I scrolled down to your comment. Though I'm not very familiar with the overall winit codebase, the design of the existing Touch event/struct is similar enough to your implementation that you could likely reuse much of what's already there and merge your other changes. I'm not too sure on the mouse or touchpad, however.

Here are my notes:

• Force's Calibrated and Normalized variants look very hacky, Calibrated having the unexpected field, altitude_angle.
• There are unecessary Options being used for both Force and Touch.