How to get faster decoding speed?

[raojay7]

Thank you for this work. The accelerate library just brings serial models and not parallel models, which brings slow decoding speed. I would like to know how to implement model parallel processing and data parallel processing like the original LLAMA code using torchrun.

[galatolofederico]

You are right, i used accelerate just to fit the big models in constrained systems. It could be a good idea to integrate something like deepspeed to and let the user decide which loading method to use