[Regression] [Bisected] Regression from Explicit Sync related patches causes Proton Games to crash gamescope

[alou-S]

Issue

• Gamescope crashes when starting Proton Games in it.
• Only occurs when using the DRM backend and doesn't occur in nested mode
• The regression occurs from commit dd77e4d for the DRM backend
• The regression occur from commit ef3ff8a on other backends

Log

[gamescope] [Info]  wlserver: [wayland] error in client communication (pid 54097)
[gamescope] [Info]  wlserver: [xwayland/server.c:217] Restarting Xwayland
XWAYLAND: wp_linux_drm_syncobj_manager_v1#9: error 0: wp_linux_drm_syncobj_surface_v1 already created for this surface
(EE) failed to dispatch Wayland events: Protocol error
[gamescope] [Info]  wlserver: [xwayland/server.c:107] Starting Xwayland on :1
[gamescope] [Error] xwm: X11 I/O error
[gamescopereaper] [Info]  reaper: Parent of gamescopereaper was killed. Killing children.
(EE) could not connect to wayland server
[gamescope] [Error] drm: drmModeRmFB failed: Bad file descriptor
[gamescope] [Error] drm: drmModeRmFB failed: Bad file descriptor
[gamescope] [Error] drm: drmModeRmFB failed: Bad file descriptor
[gamescope] [Error] drm: drmModeRmFB failed: Bad file descriptor

System Information

• Gamescope is run on a separate tty.
• Apps are started on gamescope by setting DISPLAY and WAYLAND_DISPLAY variables

System:
  Host: xeno16 Kernel: 6.10.9-1-cachyos arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 6.1.4 Distro: Arch Linux
CPU:
  Info: 8-core model: AMD Ryzen 7 5800H with Radeon Graphics bits: 64
    type: MT MCP cache: L2: 4 MiB
  Speed (MHz): avg: 1897 min/max: 400/4463 cores: 1: 3169 2: 1100 3: 1100
    4: 1100 5: 3372 6: 1100 7: 3168 8: 1100 9: 3308 10: 1100 11: 1100 12: 3168
    13: 3167 14: 1100 15: 1100 16: 1100
Graphics:
  Device-1: AMD Navi 23 [Radeon RX 6600/6600 XT/6600M] driver: amdgpu
    v: kernel
  Device-2: AMD Cezanne [Radeon Vega Series / Radeon Mobile Series]
    driver: amdgpu v: kernel
  Device-3: Quanta HP Wide Vision HD Camera driver: uvcvideo type: USB
  Display: wayland server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.2
    compositor: kwin_wayland driver: X: loaded: amdgpu,nvidia
    unloaded: modesetting dri: radeonsi gpu: amdgpu,amdgpu
    resolution: 1920x1080
  API: EGL v: 1.5 drivers: radeonsi,swrast
    platforms: gbm,wayland,x11,surfaceless,device
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.3.0-devel
    renderer: AMD Radeon Graphics (radeonsi renoir LLVM 18.1.8 DRM 3.57
    6.10.9-1-cachyos)
  API: Vulkan v: 1.3.295 drivers: radv,llvmpipe surfaces: xcb,xlib,wayland

[matte-schwartz]

I've seen this crash before on my 7900XTX, and used to be able to make it happen reliably by toggling Frame Generation in Ghost of Tsushima, but I have not seen it in a week or two since verifying I was using up to date Xwayland... which you already are 🤔 are there any games in particular that always crash, or is it all games? Also please give the exact command you're using to launch these.

[layercak3]

Does it occur if you use wined3d?

[matte-schwartz]

@alou-S Do you have a lib32-gamescope package? I was finally able to find out whythe game I was testing kept crashing when changing the window state (happened in both DRM and in nested Wayland), and it was because: 1. I had the WSI layer disabled and 2. I did not have /usr/lib32/libVkLayer_FROG_gamescope_wsi_x86.so or /usr/share/vulkan/implicit_layer.d/VkLayer_FROG_gamescope_wsi.x86.json

after changing both of those things and running the game with GAMESCOPE_ENABLE_WSI=1 gamescope -f -- %command% I have not been able to get the same XWAYLAND: wp_linux_drm_syncobj_manager_v1#9: error 0: wp_linux_drm_syncobj_surface_v1 already created for this surface crash again yet. might be worth trying for your setup and see if it makes a different with the DRM backend.

EDIT: realizing now Arch doesn't even package lib32-gamescope... you can try Valve's package since it's only the layers https://steamdeck-packages.steamos.cloud/archlinux-mirror/jupiter-main/os/x86_64/lib32-gamescope-3.14.11-1-x86_64.pkg.tar.zst and ignore the fact its an older gamescope version - shouldn't matter in this case.

[alou-S]

@matte-schwartz Firstly I don't think the lib32 layer will help since all the games I tested are 64 bit. I did install the package and it made no difference gamescope still crashes. Setting GAMESCOPE_ENABLE_WSI=1 didn't help either

[alou-S]

@layercak3 I have tested various configurations

• Linux Native games on both OpenGL and Vulkan work fine

• Games through Proton using either Vulkan or DirectX crash gamescope.
  Vulkan Games : Deadlock, Dota2
  Dx12 Games : The Finals
  Dx11 Games : Apex Legends

• Games that use dx11 when run with PROTON_USE_WINED3D prevent the crash

• Doom when run using Vulkan runs just fine which is weird.

I feel this might be a DXGI related issue since Linux Native Vulkan games run fine and running dx11 games using wined3d works fine too

[layercak3]

Also occurs with upstream wine with DXVK installed.

[alou-S]

Seems like it effects all backends since commit ef3ff8a

[matte-schwartz]

@alou-S Can you share the exact command you're using to launch these so I can see if I can repro it? I have not seen this crash in Deadlock or Dota 2 yet.

[alou-S]

gamescope --mangoapp --immediate-flip --rt and this is only when running Dota 2 on Proton

@matte-schwartz
I noticed something very weird. Dota crashes gamescope on both Vulkan and Dx11 on Proton.
Deadlock only crashes gamescope on Vulkan on Proton but Dx11 works just fine.
The Finals has a way to make it use Dx11 but it crashes gamescope on that too.

[matte-schwartz]

Played 2 matches of Deadlock earlier today with the in-game Vulkan backend and Gamescope DRM backend without crashing gamescope -e -h 2160 -w 7860 -r 240 --mangoapp --hdr-enabled -- steam -steamdeck, I'll try adding --immediate-present and --rt tomorrow.

[alou-S]

• Firstly its not a DRM backend only issue anymore it occurs in nested mode too.
• Secondly it doesn't happen mid game, it happens when the game tries to boot up.
• And finally the flags --immediate-flip and --rt have no effect on the crash. Starting gamescope without any flags causes the crash too.

[matte-schwartz]

are you sure you don't have any lingering layers or binaries that may be interfering, i.e. /etc or ~/.local instead of /usr/share? an out-of-date 64-bit Gamescope layer compared to the Gamescope binary could definitely cause something like this. also, might be helpful to see vulkaninfo --summary. are there any global environment variables set related to vulkan that you're aware of? does the game get far enough to generate a proton log if you enable it? a full gamescope output -> crash would also be helpful.

Been working here with Proton Experimental bleeding-edge branch and in-game Vulkan backend, nested gamescope on Plasma Wayland. Also tested with DRM Backend with no issues so far.

❯ gamescope --version
[gamescope] [Info]  console: gamescope version 3.15.9-1-gcc28cd7e (gcc 14.2.1)

System:
  Host: threadripper-arch Kernel: 6.10.10-2-cachyos-deckify arch: x86_64
    bits: 64
  Desktop: KDE Plasma v: 6.1.5 Distro: CachyOS
CPU:
  Info: 32-core model: AMD Ryzen Threadripper 7970X s bits: 64
    type: MT MCP MCM cache: L2: 32 MiB
  Speed (MHz): avg: 3299 min/max: 545/5673 cores: 1: 4793 2: 4789 3: 4796
    4: 2257 5: 4789 6: 2225 7: 4791 8: 4793 9: 4790 10: 2734 11: 2188 12: 2260
    13: 2188 14: 2188 15: 4772 16: 2185 17: 2196 18: 2403 19: 2188 20: 3528
    21: 2349 22: 2188 23: 2188 24: 4056 25: 4764 26: 2188 27: 2188 28: 2188
    29: 3918 30: 3122 31: 4753 32: 4663 33: 4790 34: 4793 35: 4791 36: 2219
    37: 4792 38: 4789 39: 4660 40: 3957 41: 2178 42: 2258 43: 2647 44: 2251
    45: 2232 46: 2259 47: 2261 48: 2188 49: 2188 50: 2193 51: 2869 52: 3053
    53: 2193 54: 4660 55: 2188 56: 2143 57: 4738 58: 2448 59: 4622 60: 2605
    61: 4767 62: 2554 63: 4701 64: 4757
Graphics:
  Device-1: AMD Navi 31 [Radeon RX 7900 XT/7900 XTX/7900M] driver: amdgpu
    v: kernel
  Display: wayland server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.2
    compositors: 1: Gamescope 2: kwin_wayland driver: X: loaded: modesetting
    dri: radeonsi gpu: amdgpu resolution: 5120x1440
  API: EGL v: 1.5 drivers: kms_swrast,radeonsi,swrast
    platforms: gbm,wayland,x11,surfaceless,device
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.3.0-devel
    renderer: AMD Radeon RX 7900 XTX (radeonsi navi31 LLVM 18.1.8 DRM 3.57
    6.10.10-2-cachyos-deckify)
  API: Vulkan v: 1.3.295 drivers: radv,llvmpipe surfaces: xcb,xlib,wayland

Finally, if you re-compile gamescope but change this convar:

gamescope/src/Backends/DRMBackend.cpp

Line 70 in cc28cd7

gamescope::ConVar<bool> cv_drm_debug_disable_explicit_sync( "drm_debug_disable_explicit_sync", false, "Force disable explicit sync on the DRM backend." );

to true, does gamescope work again without reverting either of the commits? it applies to both the Wayland and the DRM backends

[alou-S]

• Setting cv_drm_debug_disable_explicit_sync's default to true and compiling does mitigate the issue. This is weird since setting it via gamescopectl didn't help last time I tried.

The Vulkan layers:

Instance Layers: count = 15
---------------------------
VK_LAYER_FROG_gamescope_wsi_x86    Gamescope WSI (XWayland Bypass) Layer (x86)    1.3.221  version 1
VK_LAYER_FROG_gamescope_wsi_x86_64 Gamescope WSI (XWayland Bypass) Layer (x86_64) 1.3.221  version 1
VK_LAYER_KHRONOS_validation        Khronos Validation Layer                       1.3.290  version 1
VK_LAYER_MANGOAPP_overlay          Mangoapp Layer                                 1.3.0    version 1
VK_LAYER_MANGOAPP_overlay          Mangoapp Layer                                 1.3.0    version 1
VK_LAYER_MANGOHUD_overlay_x86      Vulkan Hud Overlay                             1.3.0    version 1
VK_LAYER_MANGOHUD_overlay_x86_64   Vulkan Hud Overlay                             1.3.0    version 1
VK_LAYER_MESA_device_select        Linux device selection layer                   1.3.211  version 1
VK_LAYER_MESA_overlay              Mesa Overlay layer                             1.3.211  version 1
VK_LAYER_OBS_vkcapture_64          OBS Linux game capture                         1.3.285  version 1
VK_LAYER_RENDERDOC_Capture         Debugging capture layer for RenderDoc          1.3.131  version 33
VK_LAYER_VALVE_steam_fossilize_32  Steam Pipeline Caching Layer                   1.3.207  version 1
VK_LAYER_VALVE_steam_fossilize_64  Steam Pipeline Caching Layer                   1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_32    Steam Overlay Layer                            1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_64    Steam Overlay Layer                            1.3.207  version 1

Logs

steam-1422450.log
gamescope.log

[matte-schwartz]

I finally see the issue you're running into (after reproing it). The reason enabling the WSI layer didn't work for you is because GAMESCOPE_ENABLE_WSI=1 is not the correct syntax, so it didn't actually enable it.

gamescope/src/steamcompmgr.cpp

Line 7345 in cc28cd7

setenv( "ENABLE_GAMESCOPE_WSI", "1", 0 );

Try ENABLE_GAMESCOPE_WSI=1 instead.

Also, regarding cv_drm_debug_disable_explicit_sync, that one still needs to be set in code during compilation afaik. Eventually I want to go through and see if I can clarify the ones that will/won't work live via gamescopectl but that's a job for another day...

[alou-S]

Even setting ENABLE_GAMESCOPE_WSI=1 didn't help :(

I can confirm that the WSI Layer is enabled since I get the following log message

[Gamescope WSI] Forcing on VK_EXT_swapchain_maintenance1.

But gamescope still crashes with the same old error.

[matte-schwartz]

do you see it creating swapchains as well, or just [Gamescope WSI] Forcing on VK_EXT_swapchain_maintenance1.

in your Proton log or gamescope output you should see:

[Gamescope WSI] Creating Gamescope surface: xid: 0x140005a
[Gamescope WSI] Atom of T was wrong type. Expected XCB_ATOM_CARDINAL.
[Gamescope WSI] Made gamescope surface for xid: 0x140005a
[Gamescope WSI] Surface state:
  steam app id:                  1422450
  window xid:                    0x140005a
  wayland surface res id:        5
  layer client flags:            0x0
  server hdr output enabled:     false
  hdr formats exposed to client: false
[Gamescope WSI] Creating swapchain for xid: 0x140005a - minImageCount: 6 - format: VK_FORMAT_R8G8B8A8_SRGB - colorspace: VK_COLOR_SPACE_SRGB_NONLINEAR_KHR - flip: true
[Gamescope WSI] Created swapchain for xid: 0x140005a - imageCount: 6

ENABLE_GAMESCOPE_WSI=0 SteamDeck=0 PROTON_LOG=1 gamescope -f -h 2160 -w 7860 -r 120 --mangoapp -- %command% reliably freezes the game for me at launch (either a black screen or right after the logo appears)

ENABLE_GAMESCOPE_WSI=1 SteamDeck=0 PROTON_LOG=1 gamescope -f -h 2160 -w 7860 -r 120 --mangoapp -- %command% works as expected here.

proton log with WSI enabled and working for reference:
steam-1422450.log

that's probably about as much as I'll be able to help here since I'm pretty much out of ideas :/ especially since I can now see what looks to be the same issue as you but I avoid it with WSI usage, which I typically have set as my default.

[alou-S]

@matte-schwartz Okay so I fixed the problem on my end too but I am not sure if its the right way to fix it.

• So like my setup has gamescope running by itself and any apps I want to run in gamescope I simply set the DISPLAY and WAYLAND_DISPLAY envars. (Mainly useful when running on a external tty via DRM backend)
• I realized setting ENABLE_GAMESCOPE_WSI=1 to the app prevents the crash but then gamescope warns me with the

CreateSwapchainKHR: Creating swapchain for non-Gamescope swapchain.
Hooking has failed somewhere!
You may have a bad Vulkan layer interfering.
Press OK to try to power through this error, or Cancel to stop.

warning at the start of the game. If I press okay and ignore the warning the game starts fine. But the question is, is this a proper way.

[KagoraTheGrindcoreAddict]

CreateSwapchainKHR: Creating swapchain for non-Gamescope swapchain.
Hooking has failed somewhere!
You may have a bad Vulkan layer interfering.
Press OK to try to power through this error, or Cancel to stop.

Currently I am also getting the same error when I run any 64-bit game under gamescope nested.

I also get this error regardless of whether I select cancel or OK:

QueuePresentKHR: Attempting to present non-hooked swapchain.
Hooking has failed somewhere!
You may have a bad Vulkan layer interfering.
Press OK to try to power through this error, or Cancel to stop.

The error does not appear for 32-bit games, but eventually both start to stutter constantly and run at 50fps after about 30 minutes.

It also crashes if I leave it unfocused for more than five minutes.

ENABLE_GAMESCOPE_WSI being to 0 or 1 makes no difference.

Distro: Artix
Gamescope version: 3.15.5
Kernel: 6.10.9
GPU: 7900XTX
Mesa: 24.3.0_devel.193759.db39685e317-1

[alou-S]

@KagoraTheGrindcoreAddict This is not a related bug. I would suggest you to create a new issue.

[matte-schwartz]

@alou-S I'm not really familiar with your specific method of launching games tbh... that error usually means you have conflicting Vulkan layers of some sort, or you're running gamescope within another instance of gamescope. Neither of which is ideal. that error probably goes away if you run embedded steam in gamescope like ENABLE_GAMESCOPE_WSI=1 gamescope -e -h <height> -w <width> -r <refresh> --mangoapp -- steam -steamdeck -gamepadui from a TTY console, and then launching games from within that gamescope-session directly. I use some variation of that command or I use Valve's gamescope-session file directly, and either method should not give that pop-up error when set up correctly.

[misyltoad]

Can you try with latest master?

[layercak3]

Thanks, I briefly checked and couldn't get this error anymore with or without the lib32 WSI. Also affects #1519

[alou-S]

@Joshua-Ashton The issue is fixed. Many thanks for the update!