Pointer offset issue when device pixel ratio is something besides "2"

[miketwenty1]

Bevy version

main branch as of commit: 425570a

This is only an issue on mobile devices when the device pixel ratio is something besides 2.

I have made a minimum viable example using one of the bevy web examples and modifying it to use the new css full screen method.

https://github.com/miketwenty1/bevy_broken_on_mobile

I'll include a video here to demo the offset issue:

2024-01-05.20-33-18.mp4

Here you can see when Pixel 7 is used with a device pixel ratio of 2.65 it has an offset from where the press is and where it corresponds. When an iPhone SE is used with a device pixel ratio of 2, it works. I assume there may be desktop devices with pixel ratios other than 2? So this may effect more than just mobile devices.

may be related to #11052

[daxpedda]

I had to go way back and remember why this is so weird.
TLDR: this should work as expected on actual devices, but won't work with simulated scale factors through dev tools.

---

The problem is that we always assume that Window.devicePixelRatio represents the amount of physical pixels one CSS pixel is. This should work as intended in Winit: we do convert CSS pixels according to Window.devicePixelRatio into physical pixels.

But as you can test yourself in the dev tools, if you change the scale factor, it only updates Window.devicePixelRatio, but it doesn't adjust CSS pixels accordingly. E.g. try window.clientWidth, which will return the same amount as with any scale factor.

The conclusion here might be that actually the size of the screen you specify in the dev tools, is just representing CSS pixels and not physical ones. Unfortunately this is not the case as can be seen when querying the physical size of the canvas:

const observer = new ResizeObserver((entries) => console.log(entries[0].devicePixelContentBoxSize[0]))
observer.observe(document.querySelector('canvas'))

This will report the same size as the size reported in CSS pixels: document.querySelector('canvas').getClientRects()[0].
So how is the CSS pixel size and the physical size the same if Window.devicePixelRatio is not 1? I'm not a CSS specification expert, but I assume this is a browser bug.

You can try adjusting the scale factor on the desktop by using the zoom function, or changing the scale factor through your OS, which will work correctly. On any mobile/tablet device that actually has a different scale factor then 1, it should work correctly as well.

Please let me know if you can reproduce this issue without the dev tools.

[miketwenty1]

@daxpedda using a physical device doesn't seem to result in the same issues!, however, I don't have a physical device that has a scale factor of 2.65 like what was happening in the dev tools.

I'm guessing I should leave this open? As having to work with physical devices for testing is big pain for development QA. Let me know your thoughts.

[daxpedda]

Unfortunately I don't think this is actionable, as there isn't really a way to detect this programmatically.

The best we can do here is report the bug to each browser.
I found a bug report at Chromium, I guess it can help to file the same for Firefox. Not sure how Safari behaves.

[miketwenty1]

@daxpedda but isn't this a regression in terms of uX from the dev point of view? this works fine with bevy 12.1

[daxpedda]

This is because Winit v0.28 didn't use ResizeObserverEntry.devicePixelContentBoxSize, so Winit had no way to actually measure the physical size of the canvas.

[miketwenty1]

I take it this means, more reliable physical device but less reliable virtual device?

[daxpedda]

I take it this means, more reliable physical device but less reliable virtual device?

Yes.
Actually this is the only API on Web that allows you to actually measure physical pixels accurately. See this demo for an example of what kind of errors this prevents.