CUDA not using the GPU

[veczieds]

Hello!
I set up a virtual machine with a Tesla T4 GPU so i can do the training on that machine. But i cant make the training utilize the GPU. In the config file CUDA is set to true. I even tried running competition script with "cuda:0" listed as the device to use for training but nothing. All the drivers seem to be configured correctly. I even tried to use WDDM and TCC drivers but nothing with either of them. nvidia-smi doesnt show any python processes running on the card. Maybe someone could point where to look.
Just quickly wanted to add that this project is amazing. Im not too familiar with machine learning, but project creators have made it easy to get into this stuff.

[yannbouteiller]

Hi, thanks! :)

Probably your issue comes from the fact you are using a virtual machine?

On my laptop, running both the worker (with trackmania) and the trainer (with CUDA enabled), the trainer uses CUDA alright:

(this is Windows but I usually have the trainer run remotely on a Linux machine)

Probably you didn't miss that, but just to double-check, in the config.json file that is on your trainer machine you want something like this:

"CUDA_TRAINING": true,
"CUDA_INFERENCE": false,

(Inference is for the worker, training is for the trainer)

Maybe on your virtual machine you can start by double-checking that pytorch finds CUDA:

(tmrl) C:\Users\yannb>python
Python 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

[veczieds]

I did some more reading and already checked if pytorch finds CUDA. And it doesnt. Im trying to look for a reason why it cant find CUDA as i have the compatible versions. At the moment i installed CUDA 11.8 cause Pytorch website showed pytorch installation for 11.8.

[yannbouteiller]

Yeah that's you issue then. If you can, I recommend using anaconda and their CUDA version of pytorch, this usually fixes this kind of issues (although not sure on a VM):

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

[veczieds]

I avoided using anaconda. Will try this and come back with results

[veczieds]

Looks like it solved the problem. Thank you very much!

[yannbouteiller]

Great! You're welcome :)