-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Skip thinking section in Claude tool call response #226
base: main
Are you sure you want to change the base?
Conversation
|
Not sure if it's related, but I'm also unable to manually handle function calls (although as you'll see below, it would also likely apply to from magentic import ParallelFunctionCall, StreamedStr, prompt, prompt_chain, chatprompt, SystemMessage, UserMessage, AssistantMessage, FunctionResultMessage, FunctionCall
from magentic.chat_model.anthropic_chat_model import AnthropicChatModel
def get_weather(city: str) -> str:
return f"The weather in {city} is 20°C."
function_call = FunctionCall(function=get_weather, city="Cape Town")
messages = [
SystemMessage("You are helpful."),
UserMessage("What's the weather like in Cape Town?"),
AssistantMessage(function_call),
FunctionResultMessage(function_call=function_call, content="The weather in Cape Town is 20°C.")
]
@chatprompt(
*messages,
functions=[get_weather],
model=AnthropicChatModel(
model="claude-3-opus-20240229",
temperature=0.2,
)
)
def _llm() -> FunctionCall | StreamedStr: ...
response = _llm()
response Which leads to the following Anthropic API error:
What's bizarre is that if I modify the way the function results get handled, to return a string, rather than converting into an object / dict: @message_to_anthropic_message.register(FunctionResultMessage)
def _(message: FunctionResultMessage[Any]) -> ToolsBetaMessageParam:
function_schema = function_schema_for_type(type(message.content))
return {
"role": AnthropicMessageRole.USER.value,
"content": [
{
"type": "tool_result",
"tool_use_id": message.function_call._unique_id,
"content": function_schema.serialize_args(message.content) #json.loads(function_schema.serialize_args(message.content)),
}
],
} Then it works. The Anthropic docs seems to suggest that the answers can now be specified as a string, or as a list of nested content blocks. I'm wondering if this changed with the public beta of tool use? Either way, it seems to work this way now. Would probably be a good test case to include in the future! |
Skip the
<thinking>
/ chain-of-thought section when parsing the response from Claude.This seems to always be included when tools are provided in the request.The models do not reliably start their answers with the<thinking>
tag, so the changes here are only a partial fix for this. Seems like the only full solution is to not use streaming responses so the whole response can inform how to parse.This is only an issue for return type annotations that union
str
/StreamedStr
with a structured object orFunctionCall
/ParallelFunctionCall
, because the potential for string output means a tool call cannot be forced.Potential full solution: if a union of tool call and string return type is given, then stream the whole response before determining how to parse it. This would essentially disable streaming in this case, which means
StreamedStr
/Iterable[T]
/ParallelFunctionCall
would arrive all at once instead of as the parts are generated.Issue #220
In future, an
AnthropicAssistantMessage
could be added with athinking: str
attribute. This could be registered withmessage_to_anthropic_message
so thethinking
string is persisted and serialized back to the model. This could be exposed to users for use in@chatprompt
(non-anthropic models would treat it likeAssistantMessage
and ignorethinking
).