Repository navigation
The stop parameter in openai API doesn't work since v0.2.5 #1048
Description
Since version v0.2.5, it seems the stop parameter in openai api is directly set conv.stop_str, rather than from request.
https://github.com/lm-sys/FastChat/blob/v0.2.5/fastchat/serve/api.py#L134
In version v0.2.3, it works when set in the request.
https://github.com/lm-sys/FastChat/blob/v0.2.3/fastchat/serve/api.py#L125
The stop parameter is a key when it works with ReAct in langchain, seems quite important to enable.
Activity
merrymercy commentedon May 8, 2023
Thanks for reporting this. Could you send a pull request to fix it?
jstzwj commentedon May 8, 2023
Fixed in #818.
llama-assistant commentedon May 9, 2023
@jstzwj Thanks for the fix, happy to see it in next version.
mingfang commentedon May 19, 2023
In the openai_api_server, stop works for non-streaming completions, but not for streaming.
The problem is the unwanted stop sequence gets streamed out before stopping.
https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L518
As a result, this breaks LangChain ReAct agents
merrymercy commentedon May 20, 2023
@andy-yang-1 does the new PR (#1246) fix this?
@mingfang If not, could you contribute a PR to fix it?
mingfang commentedon May 20, 2023
I tested the PR locally and it has the same problem.
@merrymercy do you think this problem should be fixed in
https://github.com/andy-yang-1/FastChat/blob/langchain-support/fastchat/serve/inference.py#L51
so that it doesn't emit the stop sequence?
andy-yang-1 commentedon May 20, 2023
@merrymercy My PR didn't fix the problem, how can we solve it?
merrymercy commentedon May 20, 2023
We handle the stop string here https://github.com/andy-yang-1/FastChat/blob/fae4087bbb6f7979b61f2e0c2912d77547a5c659/fastchat/serve/inference.py#L164-L175,
I think it will correctly delete the stop sequence finally. Does it occur during the middle of the streaming?
mingfang commentedon May 20, 2023
The problem happens when the previous generate token is the partial beginning of the stop sequence.
It will not match the entire stop sequence until the next few tokens.
As a result the partial stop sequence is stream the client, causing ReAct to fail.
mingfang commentedon May 20, 2023
@merrymercy
This is my PR with the stop detection fix
#1392
Chessing234 commentedon Jun 2, 2026
The partial-stop suffix check is fixed in #3883 so streaming should no longer emit prefixes like
Obsbefore the full stop string.