Enable ROCM in CI #999

msaroufim · 2024-10-03T01:05:30Z

Salient points:

Add ROCm nightly wheels to test matrix
Refactor conda-builder -> almalinux-builder pytorch#140157
The above PR shows that we've migrated to almalinux-builder due to the EOL CENTOS 7. Changes to regression_test.yml to not install devtoolset-10 have been made in accordance with this switch.
Fix bug in torchao/utils.py in invocation of torch.cuda.get_device_properties()

Needs changes in pytorch/test-infra#6104

pytorch-bot · 2024-10-03T01:05:33Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/999

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit bde89de with merge base 63d142c ():

NEW FAILURES - The following jobs have failed:

Run Regression Tests / test (CUDA 2.4, linux.g5.12xlarge.nvidia.gpu, torch==2.4.0, cuda, 12.1) / linux-job (gh)
test/quantization/test_quant_api.py::TestQuantFlow::test_quantized_tensor_subclass_int8_dyn_quant
Run Regression Tests / test (CUDA 2.5.1, linux.g5.12xlarge.nvidia.gpu, torch==2.5.1 --index-url https://download.pytorch... / linux-job (gh)
test/quantization/test_quant_api.py::TestQuantFlow::test_quantized_tensor_subclass_int8_dyn_quant
Run Regression Tests / test-nightly (ROCM Nightly, linux.rocm.gpu, --pre torch --index-url https://download.pytorch.org/... / linux-job (gh)
test/sparsity/test_marlin.py::SparseMarlin24::test_quant_sparse_marlin_layout_eager

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2024-10-03T01:05:36Z

No ciflow labels are configured for this repo.
For information on how to enable CIFlow bot see this wiki

msaroufim · 2024-10-09T01:22:50Z

@atalman im not sure the no-sudo flag does anything. Tried a few variants for the value like true or "true" and same result

.github/workflows/regression_test.yml

amdfaa · 2024-11-19T15:49:37Z

@pytorchbot rebase

jithunnair-amd · 2024-12-19T22:26:16Z

.github/workflows/regression_test.yml

+            torch-spec: '--pre torch --index-url https://download.pytorch.org/whl/nightly/rocm6.2.4'
+            gpu-arch-type: "rocm"
+            gpu-arch-version: "6.2.4"
+


Move this section to the test-nightly job since that's where we are testing the nightly wheels, and it uses linux_job_v2.yml, which should be updated as per pytorch/test-infra#6003 (comment)

jithunnair-amd · 2024-12-19T22:28:10Z

.github/workflows/regression_test.yml

    with:
      timeout: 120
+      no-sudo: ${{ matrix.gpu-arch-type == 'rocm' }}
+      rocm: ${{ matrix.gpu-arch-type == 'rocm' }}
+      continue-on-error: ${{ matrix.gpu-arch-type == 'rocm' }}


This should definitely not be checked-in, since it's only for us to gather a complete list of test failures. @msaroufim Would we merge this PR only after ROCm CI is fully clean? I'd rather get all these infra changes merged, so that we run torchao CI on ROCm regularly, and maybe skip any failing tests for ROCm while we work separately to enable them.

It's up to you, the main constraint is we can't really be having CI per commit or on main run red since then it just causes confusion and people slowly learn to ignore seeing red. So if you'd like to merge some variant of this PR without running on commits to or on main then we can try to merge this more quickly

Personally I'd favor merging the skip tests as part of this work and we can do enablement for tests one by one easily while maintaining a green CI

@petrex Please note that torchao team would like to have this PR be merged with a clean signal for ROCm, so please skip any failing tests as part of this PR

Needed for pytorch/test-infra#6003 and pytorch/ao#999 Pull Request resolved: #143590 Approved by: https://github.com/atalman Co-authored-by: Jithun Nair <[email protected]>

petrex · 2025-01-06T20:59:48Z

happy new year @jithunnair-amd @amdfaa Is this feature/PR ready to deploy?

jithunnair-amd · 2025-01-08T08:52:09Z

happy new year @jithunnair-amd @amdfaa Is this feature/PR ready to deploy?

2 pending items:

Resolve issue with aws credentials causing some steps to fail in CI job eg. https://github.com/pytorch/ao/actions/runs/12656214677/job/35268262299#step:17:39 : this one needs pytorch infra team to enable "AWS trust policy" for torchao repository cc @huydhn to submit a PR for that
Skip any failing unit tests for ROCm as part of this PR as per Enable ROCM in CI #999 (comment) cc @petrex to add skips to this PR

huydhn · 2025-01-08T21:50:11Z

1. Resolve issue with aws credentials causing some steps to fail in CI job eg. https://github.com/pytorch/ao/actions/runs/12656214677/job/35268262299#step:17:39 : this one needs pytorch infra team to enable "AWS trust policy" for torchao repository cc @huydhn to submit a PR for that

The credential is working now. There is a new failure w.r.t chown on the CI job https://github.com/pytorch/ao/actions/runs/12656214677/job/35334719646, but it’s a different story I think

Enable ROCM in CI

e3839b6

pytorch-bot bot added ciflow/rocm module: rocm labels Oct 3, 2024

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 3, 2024

msaroufim added 5 commits October 3, 2024 14:48

try a different label

d9bf84c

Merge branch 'main' into msaroufim-patch-21

760a6f4

Update regression_test.yml

08ff253

Update regression_test.yml

a3c480e

Update regression_test.yml

031ea5c

atalman reviewed Oct 9, 2024

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

Update .github/workflows/regression_test.yml

484a5d5

atalman reviewed Oct 10, 2024

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

Update .github/workflows/regression_test.yml

af2899b

atalman reviewed Oct 10, 2024

View reviewed changes

.github/workflows/regression_test.yml Outdated Show resolved Hide resolved

atalman and others added 2 commits October 10, 2024 08:58

Update .github/workflows/regression_test.yml

46b9afd

Merge branch 'main' into msaroufim-patch-21

b282d74

This was referenced Oct 15, 2024

Enable Intel GPU #753

Closed

Multibackend tracker #1082

Open

atalman mentioned this pull request Oct 15, 2024

Add support for Rocm Runners to linux_job pytorch/test-infra#5766

Open

jithunnair-amd and others added 7 commits October 22, 2024 18:55

Retrigger CI

7541194

Merge branch 'main' into msaroufim-patch-21

fda91b3

Update regression_test.yml

0f9b665

Update regression_test.yml

4757c27

Update regression_test.yml

9f15702

Merge branch 'main' into msaroufim-patch-21

58454c2

fixed

6f0751d

amdfaa and others added 15 commits December 6, 2024 12:04

Update regression_test.yml

cc51718

trigger rebuild

775322c

trigger rebuild

15b29a6

trigger rebuild

e0803f9

trigger rebuild

7980247

trigger rebuild

6453e1d

trigger rebuild

91abfba

trigger rebuild

a882e01

Provide GPU index to get_device_properties()

ed2412b

trigger rebuild

d434f13

trigger rebuild

859fbcd

Update regression_test.yml

ee7cde6

Update regression_test.yml

1155deb

Lint

aa63d83

Use latest value of timeout in upstream

8834b4c

jithunnair-amd mentioned this pull request Dec 19, 2024

[ROCm] Update setup-rocm for almalinux-based images pytorch/pytorch#143590

Closed

jithunnair-amd requested changes Dec 19, 2024

View reviewed changes

Update regression_test.yml

eba433b

amdfaa added 3 commits December 23, 2024 16:21

trigger rebuild

e71c7ce

trigger rebuild

f2df988

Update regression_test.yml

b2f2f8d

amdfaa mentioned this pull request Dec 23, 2024

Enable linux_job_v2.yml workflow for ROCm pytorch/test-infra#6104

Draft

trigger rebuild

ed3b240

Update regression_test.yml

cfe216c

jithunnair-amd and others added 2 commits January 8, 2025 08:53

Retrigger CI

f1ec235

Update to rocm6.3 nightly wheels

bde89de

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable ROCM in CI #999

Enable ROCM in CI #999

msaroufim commented Oct 3, 2024 •

edited by jithunnair-amd

Loading

pytorch-bot bot commented Oct 3, 2024 •

edited

Loading

pytorch-bot bot commented Oct 3, 2024

msaroufim commented Oct 9, 2024

amdfaa commented Nov 19, 2024

jithunnair-amd Dec 19, 2024

jithunnair-amd Dec 19, 2024

msaroufim Dec 20, 2024 •

edited

Loading

jithunnair-amd Jan 8, 2025

petrex commented Jan 6, 2025

jithunnair-amd commented Jan 8, 2025

huydhn commented Jan 8, 2025

Enable ROCM in CI #999

Are you sure you want to change the base?

Enable ROCM in CI #999

Conversation

msaroufim commented Oct 3, 2024 • edited by jithunnair-amd Loading

pytorch-bot bot commented Oct 3, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/999

❌ 3 New Failures

pytorch-bot bot commented Oct 3, 2024

msaroufim commented Oct 9, 2024

amdfaa commented Nov 19, 2024

jithunnair-amd Dec 19, 2024

Choose a reason for hiding this comment

jithunnair-amd Dec 19, 2024

Choose a reason for hiding this comment

msaroufim Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

jithunnair-amd Jan 8, 2025

Choose a reason for hiding this comment

petrex commented Jan 6, 2025

jithunnair-amd commented Jan 8, 2025

huydhn commented Jan 8, 2025

msaroufim commented Oct 3, 2024 •

edited by jithunnair-amd

Loading

pytorch-bot bot commented Oct 3, 2024 •

edited

Loading

msaroufim Dec 20, 2024 •

edited

Loading