feat: Scanning GPU allocation map

[jopemachine]

Resolves #3327. (https://github.com/lablup/giftbox/issues/638) (BA-428) (GF-67).

Implement an API that allows the administrator to check how fGPU is allocated among agents through GPU alloc map (GPU allocation states per GPU device).

How it work

The GPU allocation is calculated by reading the resource.txt file in the scratch directory per kernel and summing up the allocation information in KernelResourceSpec.

Usage example

Note

This describes the same issue addressed in issue #638.

Tested using mock-accelerator.

Here is a simple example with which we can test this PR.

When I specify below two mock GPU devices in mock-accelerator.toml, I have 2 fGPUs in total.

devices = [
  { mother_uuid = "c59395cd-ac91-4cd3-a1b0-3d2568aa2d01", model_name = "CUDA GPU", numa_node = 0, subproc_count = 108, memory_size = "2G", is_mig_device = false },
  { mother_uuid = "c59395cd-ac91-4cd3-a1b0-3d2568aa2d02", model_name = "CUDA GPU", numa_node = 1, subproc_count = 108, memory_size = "2G", is_mig_device = false },
]

And after creating session like below command,

❯ ./backend.ai session create \
            -r cpu=1 -r mem=2g -r cuda.shares=0.2 \
            cr.backend.ai/testing/ngc-pytorch:23.10-pytorch2.1-py310-cuda12.2
∙ Session ID e114540d-bd7e-4765-bb25-4b00a47feb51 is created and ready.
∙ This session provides the following app services: sshd, ttyd, jupyter, jupyterlab, vscode, tensorboard, mlflow-ui, nniboard

❯ ./backend.ai session create \
            -r cpu=1 -r mem=2g -r cuda.shares=1.2 \
            cr.backend.ai/testing/ngc-pytorch:23.10-pytorch2.1-py310-cuda12.2
∙ Session ID 91bf45c7-43f3-4c52-9e49-48d49bc897f7 is created and ready.
∙ This session provides the following app services: sshd, ttyd, jupyter, jupyterlab, vscode, tensorboard, mlflow-ui, nniboard

I can query the gpu_alloc_map as json format using the following query statement.

query ($agent_id: String!) {
  agent(agent_id: $agent_id) {
    gpu_alloc_map
  }
}

{
  "data": {
    "agent": {
      "gpu_alloc_map": "{\"c59395cd-ac91-4cd3-a1b0-3d2568aa2d02\": \"0.80\", \"c59395cd-ac91-4cd3-a1b0-3d2568aa2d01\": \"0.60\"}"
    }
  }
}

And we can see two mock GPU devices have been allocated 0.6 and 0.8 fGPU respectively.

In the first request, 0.2 fGPU was allocated to the second GPU.
Since there was no device available to allocate 1.2 fGPU in the second request, it can be seen that 0.6 fGPU was evenly distributed and allocated to both GPU devices.

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue
• API server-client counterparts (e.g., manager API -> client SDK)

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “flow:merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “flow:hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[jopemachine]

• feat: Cache gpu_alloc_map in Redis, and Add RescanGPUAllocMaps mutation #3293
• feat: Scanning GPU allocation map #2273 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[HyeockJinKim]

�lgtm

[HyeockJinKim]

�lgtm