Getting Started with RenderDoc
RenderDoc is a graphics debugger that lets you capture a single frame and analyze how it's rendered. It's useful when developing mods that involve material properties, new models, or shaders. It's also just fun to see how things work.
Once you've installed RenderDoc, the first step is getting it to attach itself to DSP. You can do this either by having RenderDoc launch the game directly or by "injecting" it into a running process.
Pros: Easy
Cons: Skips loading BepInEx/mods
If you don't launch DSP through Steam, it crashes before it starts loading. Get around this by adding a steam_appid.txt file in the same directory as DSPGAME.exe. The txt file should contain 1366540 and nothing else. You can test this by just opening DSPGAME.exe directly. If you make it to the main menu, you're good.
In RenderDoc, either switch to the Launch Application tab or get through File>Launch Application. Set the Executable Path to DSPGAME.exe and click Launch.
Pros: Works with BepInEx/Mods
Cons: Annoying
This feature is not enabled by default and must be enabled in RenderDoc. In Tools>Settings, check the box for "Enable Process Injection". Restart RenderDoc.
For injection to work, it has to happen before any rendering starts. Getting that timing right is probably impossible, but we can have DSP launch and wait before starting. Edit boot.config in the DSPGAME_Data directory and set wait-for-native-debugger = 1. Now when you launch DSP, you'll get a pop up before it launches saying "You can attach a native debugger now if you want". Yes, please.
Open DSPGame.exe, and once that pop up shows, switch to RenderDoc. File>Inject Into Process. You'll see DSPGame.exe on the list with the window title "Debug". Select and click the inject button. Now you can click OK on the pop up and proceed as normal.
If you see RenderDoc related info at the top left of the screen, it's working. Play the game as normal until you get what you want to capture on screen. If you're just trying this out, this is probably the scene just after starting a new game once the mecha is on the planet and controllable.
Hit F12 to capture the frame. You can do this multiple times if desired. Once you've captured a frame. You can exit DSP and return to RenderDoc. You should see your capture(s) shown in a list. Open the on you're interested in.
You should see a screen similar to this.
The Event Browser on the left lets you step through each draw (or similar) call. As you move down through events, the Texture Viewer tab will show the scene as it is drawn for each draw call. If you want to skip to the interesting stuff, open the Colour Pass #5 section.
With a Draw event selected, the Pipeline State tab will show each step in the rendering pipeline. The most interesting steps being the Vertex Shader and Pixel Shader steps. This will show buffers/textures attached, constant buffers, and the name of the shader used, among other details. Next to the shader name, you can even view the semi-decompiled source for the shader.
With a Draw event selected, the Mesh Viewer tab will show the actual mesh being rendered in wireframe. Also shows the variables passed per vertex such as its position, normals, tangents, and UVs. Also shows the output of the vertex shader on the right.
