Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inference stop_str missing filter fix #2792

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Trangle
Copy link
Contributor

@Trangle Trangle commented Dec 9, 2023

Why are these changes needed?

  1. Fro stop_str = ["</s>", "<|im_end|>"], although "<| im_ end |>" has been added to "special_tokens", there some cases like
I am a chatbot created by researchers.<
I am a chatbot created by researchers.<|
I am a chatbot created by researchers.<|im_
I am a chatbot created by researchers.<|im_end
I am a chatbot created by researchers.<|im_end|><
I am a chatbot created by researchers.<|im_end|><|im_end|><

The original logic will only find the last<| im_end|> through rfind, which in turn leads to filtering failure.
2. When no stop is generated and "rfind_start" is not updated, the starting position is always found, which is time-consuming.
3. Determine the search subset based on the maximum "stop_token" length to improve search efficiency.

Related issue number (if applicable)

Checks

  • I've run format.sh to lint the changes in this PR.
  • I've included any doc changes needed.
  • I've made sure the relevant tests are passing (if applicable).

@Trangle
Copy link
Contributor Author

Trangle commented Jan 17, 2024

@merrymercy Help deal this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants