Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi-gpu inference #704

Open
cdfan0627 opened this issue Feb 11, 2025 · 4 comments
Open

multi-gpu inference #704

cdfan0627 opened this issue Feb 11, 2025 · 4 comments
Assignees
@zRzRzRzRzRzRzR

Comments

@cdfan0627
Copy link

請問有支援多顆gpu的inference嗎，可以解決單顆gpu會out of memory的問題
@zRzRzRzRzRzRzR zRzRzRzRzRzRzR self-assigned this Feb 14, 2025
@zRzRzRzRzRzRzR
Copy link
Member

支持的，在我们的cli 中，允许多卡推理，请查看cli_demo.py

@cdfan0627
Copy link
Author

可以請問是在 cli_demo.py 哪行寫了多卡推理嗎？

@zRzRzRzRzRzRzR
Copy link
Member

Image

pipe = CogVideoXImageToVideoPipeline.from_pretrained(model_path, torch_dtype=dtype,device_map="balanced")

@zRzRzRzRzRzRzR
Copy link
Member

pipe.to("cuda")

保持注释，因为这会再次将pipe放到同一张GPU。

需要注意的是，即使切割成多个GPU，每个GPU最少也需要16G显存

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zRzRzRzRzRzRzR @cdfan0627