Will TensorRT-LLM be available within Triton or will it be a separate server ? #6290
-
|
Like many, I am pretty stoked about the TensorRT-LLM announcement. I am wondering if this will be accessible from within Triton as a specific backend or will we need to run a separate process to benefit from TensorRT-LLM ? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Very happy to hear that, Matthieu! Thanks for sharing. Triton will continue to work as the one solution for all of your AI model inferencing. We are working on a TensorRT-LLM backend that you can easily plug your models into. |
Beta Was this translation helpful? Give feedback.
-
|
Amazing !! Thank you for your answer |
Beta Was this translation helpful? Give feedback.
Very happy to hear that, Matthieu! Thanks for sharing.
Triton will continue to work as the one solution for all of your AI model inferencing. We are working on a TensorRT-LLM backend that you can easily plug your models into.