Wrong scaling in fullscreen VLC (was: Problems playing videos in VLC)

[jtyr]

After upgrading libvdpau-va-gl from 0.3.4 to 0.3.6, I'm experiencing problems with playing videos in VLC. The video doesn't scale to full screen size when in the full screen mode and some videos have wrong aspect ratio. When I downgrade back to libvdpau-va-gl-0.3.4, all works normally.

VLC: 2.2.2
Distro: Arch Linux

[stefano-m]

👍 Same issue on ArchLinux for me.

If I build version 0.3.5 from Arch[1] everything looks fine again.

[1] https://projects.archlinux.org/svntogit/community.git/commit/trunk?h=packages/libvdpau-va-gl&id=7e408c65e3e60b7270600082a2b790290ac83de8

[soimort]

A quick bisect indicates that the issue was caused by commit e10ef7a.

For VDPAU backend, VLC checks for the value of vdp_video_mixer_query_parameter_value_range. As it used to return the "unimplemented state" VDP_STATUS_NO_IMPLEMENTATION, VLC simply falls back to normal mode thus not actually using libvdpau-va-gl (any version before 0.3.6) at all. [1]

However, since this commit, VLC starts to render through libvdpau-va-gl and the issue exposes.

P.S. mpv seems to play well with libvdpau-va-gl 0.3.6, so I wonder if it's an issue of VLC's VDPAU module implementation rather than a bug of this driver.

[1] https://github.com/videolan/vlc/blob/2.2.0-git/modules/hw/vdpau/display.c#L523

[promeneur]

some info about vlc and "va-api x11"
if you select "automatic" for input/codecs > hardwaredecoding parameter
then
even if your graphic adapter is an intel one and your driver is i965, vlc tries to uses vdpau, fails and does not uses va-api !

if you select "va-api via x11" then vlc uses va-api

i sent a bug report about this but the answer is "it is not a bug"

so it is necessary to install libvdpau-va-gl for this pb. no ?

[promeneur]

replace
"so it is necessary to install libvdpau-va-gl for this pb. no ?"
by
"so it is not necessary to install libvdpau-va-gl for vlc getting hw accel decoding. no ?"

[i-rinat]

Hi.
There were very similar issue #50. There I found that VLC 2.2.2 (at least the version from Debian repository) tried to use VDPAU API in a non-recommended way: trying to use deleted resource. Bug was triggered by a specific video sizes and were dormant until libavcodec changed its behavior a bit.

Since then, there was a 2.2.3 release, and also recently 2.2.4. Release notes for 2.2.3 have VDPAU fixes mentioned. And indeed in 2.2.3 currently available in Debian testing, I see VLC calling VDPAU functions for decoding and rendering.

@jtyr, @stefano-m, @soimort, @promeneur, could you check with a newer version of VLC, please? 2.2.3 and later should be fine.

[i-rinat]

A quick bisect indicates that the issue was caused by commit e10ef7a.

Before that commit VLC just ignored libvdpau-va-gl. It asked for supported dimensions, saw return code, and gave up with VDPAU. As far as I remember, I added the patch to check if the same issue (as current one) can be reproduced with VLC from master branch.

[vincentbernat]

I still have the problem with vlc 2.2.4 in unstable. If you want me to trace through some functions with gdb, I can.

[promeneur]

opensuse 13.1 x86_64
vlc 2.2.4
libvdpau_va_gl1 0.3.6

same problem with scaling picture to full screen.

[i-rinat]

Oh. I was testing the other issue — general playback failure in VLC with libvdpau-va-gl. Now when I read description again, I see it's about fullscreen mode.

Now I can see the bug on my machine too.

[i-rinat]

Also tested v0.3.4 (all with code added to vdpVideoMixerQueryParameterValueRange() to make VLC use libvdpau-va-gl), v0.3.3, v0.3.2, and v0.3.0. Bug is there, so it wasn't introduced in v0.3.6, only revealed in that version. Will keep debugging further.

[i-rinat]

Well, I'm confused.

Here is one of the calls that VLC make:

vdp_video_mixer_render(300052, 4294967295, NULL, 2, 2, {300015, 300009}, 300010, 1, {300011}, {0, 0, 854, 480}, 300049, {0, 0, 1366, 768}, NULL, 0, NULL)
    -> 0

There are many parameters for the function, but the most interesting ones are the last four. They are:

[in]    destination_rect    The sub-rectangle of the destination surface to modify. Note that rectangle clips all other actions.
[in]    destination_video_rect  The sub-rectangle of the destination surface that will contain the processed video. This rectangle is relative to the entire destination surface. This rectangle is clipped by destination_rect. If NULL, the destination rectangle will be sized to match the source rectangle, and will be located at the origin.
[in]    layer_count The number of additional layers to composite above the video.
[in]    layers  The array of additional layers to composite above the video. 

So here destination_rect is set to {0, 0, 1366, 768} (1366x768 is my fullscreen resolution), and destination_video_rect is set to NULL which means — take source rectangle, which is {0, 0, 854, 480}. As far as I can tell, correct interpretation of the call is: "take 854x480 video and render it to 1366x768 source into 854x480 rectangle located at top left corner".

Since authors of VDPAU code in VLC must have been tested it, I assume it works fine on nVidia proprietary drivers. But I wonder, how. Haven't chance yet to test it on machine with nVidia adapter.

[i-rinat]

For another example, here is how mpv makes the same call:

vdp_video_mixer_render(300010, 4294967295, NULL, 2, 2, {4294967295, 4294967295}, 300012, 2, {4294967295, 4294967295}, {0, 0, 854, 480}, 300014, NULL, {0, 0, 1366, 767}, 0, NULL)
    -> 0

Here destination_rect is NULL, which means: "full destination surface size", 1366x768 in this case. But destination_video_rect is {0, 0, 1366, 767}, which means: "scale video to 1366x767".

libvdpau-va-gl scales video fine with mpv.

[soimort]

I can confirm the same issue on an nVidia GeForce machine, as well as on an Intel one, both running

• Arch Linux x86_64
• vlc 2.2.4-1
• libvdpau-va-gl 0.3.6-1
• libvdpau 1.1.1-2
• libva 1.7.0-1
• mesa-libgl 11.2.2-1

mpv works perfectly fine with libvdpau-va-gl.

[i-rinat]

@soimort, could you try to build VLC with the following patch?

diff --git a/modules/hw/vdpau/chroma.c b/modules/hw/vdpau/chroma.c
index 1fe511c..630ad89 100644
--- a/modules/hw/vdpau/chroma.c
+++ b/modules/hw/vdpau/chroma.c
@@ -642,7 +642,7 @@ static picture_t *VideoRender(filter_t *filter, picture_t *src)
     err = vdp_video_mixer_render(sys->vdp, sys->mixer, VDP_INVALID_HANDLE,
                                  NULL, structure,
                                  MAX_PAST, past, surface, MAX_FUTURE, future,
-                                 &src_rect, output, &dst_rect, NULL, 0, NULL);
+                                 &src_rect, output, NULL, &dst_rect, 0, NULL);
     if (err != VDP_STATUS_OK)
     {
         msg_Err(filter, "video %s %s failure: %s", "mixer", "rendering",

I think, it should fix video scaling.

[soimort]

That does fix the scaling issue for me. 👍

[i-rinat]

@soimort, what video adapter are you trying on? And what driver version?

I've just tested on GeForce GTX 980 with 352.79-8 (Debian testing) with VLC 2.2.4. It scales video fine. That confuses me more, since I've seen other people with nVidia adapters writing about fullscreen scaling issue on the Internet.

[soimort]

GeForce 940M (with Intel integrated graphics) via Bumblebee. Driver & kernel version:

• nvidia 367.27-1
• linux 4.6.2-1

The issue was not just about fullscreen mode; videos simply fail to scale accordingly as the window size changes. Applying the patch above should fix the issue (on both intel and NVIDIA).

[i-rinat]

Bumblebee

That means you are using only Intel for video decoding, as Bumblebee doesn't pass VDPAU calls to nVidia adapter. It only works for OpenGL.

[soimort]

Then I have no idea about whether this is an issue on nVidia. (just realized Bumblebee doesn't support VDPAU yet)

Any chance you could send the patch to VLC? It seems to me that this bug is on the VLC side.

[i-rinat]

Any chance you could send the patch to VLC?

Sent already, yesterday. 😃
And was told that there is a possible misinterpretation in that particular API entry description. In that case the correct version is how nVidia proprietary drivers do. According to my tests, VLC scales video fine on nVidia proprietary drivers.

Need some time for clarifications. But most probably, it will be fixed in libvdpau-va-gl. I just want to be sure about changes.

[i-rinat]

Letter with a question was also redirected to VDPAU list, but VDPAU list is still silent to the moment.

But most probably, it will be fixed in libvdpau-va-gl

Pushed a fix into master branch as fddcb50. And also as a patch for v0.3.6: 5e193ed.

[akostadinov]

The issue does happen for me on intel video (without nvidia card at all/as somebody already mentioned video output to "auto"). Setting output to "opengl" fixes the scaling issue. To me it seems VLC is using vdpau wrapper for intel card and scaling doesn't work. More likely vdpau/vlc issue than nvidia driver issue.

[i-rinat]

Contrary to (my) expectations, patch was pushed to the VLC code too (videolan/vlc@7df48e3). Perhaps, for compatibility reasons.

Unfortunately, VDPAU mailing list still looks abandoned. No replies, no clarifications.

[promeneur]

if it can help.

intel graphical adapter

if setting is "automatic" for "hardware decoding"
i confirm that vlc launches vdpau then vdpau launches va-api.

it's not robust. if file is not very clean then vlc crashes.
vlc never crashes if settings is "va-api via x11" for "hardware decoding"