Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [GenAI-Perf] openai-fronted server with --endpoint-type completions #7995

Open
jihyeonRyu opened this issue Feb 7, 2025 · 0 comments
Labels
openai OpenAI related

Comments

@jihyeonRyu
Copy link

Description
I’m currently running perf-analyzer with the --endpoint-type completions option. But then I encountered the following error:

Thread [0] had error: OpenAI response returns HTTP code 400: {"detail":"only single string input is supported"}

Could you please advise on how to resolve this issue?

*The curl to completions has no problem.
*Only genai-perf has problem.

Triton Information
What version of Triton are you using?

Triton Server Version 2.54.0
TensoRT-LLM v0.17.0
genai-perf 0.0.10 (PIP)
server/python/openai: main branch (18a9c51)

Are you using the Triton container or did you build it yourself?

To Reproduce
Steps to reproduce the behavior.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

Expected behavior
A clear and concise description of what you expected to happen.

@rmccorm4 rmccorm4 added question Further information is requested openai OpenAI related and removed question Further information is requested labels Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
openai OpenAI related
Development

No branches or pull requests

2 participants