Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Parsing of tool calls does not support arrays in the "arguments" #3388

Open
bwilliams2 opened this issue Jun 7, 2024 · 0 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@bwilliams2
Copy link
Contributor

bwilliams2 commented Jun 7, 2024

Describe the bug
The current regex pattern used for parsing tool calls results in an incomplete parsing when nested objects are used within the tool call definition. This severely limits the usefulness of tool calls within Prompty since any arguments with nested objects break parsing.

Relevant code:

pattern = r"\n*#{0,2}\s*tool_calls\s*:\s*\n+\s*(\[.*?\])"
match = re.search(pattern, role_prompt, re.DOTALL)

How To Reproduce the bug
Successful parsing

from promptflow.core._prompty_utils import try_parse_tool_calls

role_prompt = \
"""
## tool_calls: 
[{
    "id": "call_dks134",
    "type": "function",
    "function": { 
        "name":"update",
        "arguments": '{"items": {"name": "test_name"}}'
    }
}]
"""
success = try_parse_tool_calls(role_prompt)

Match group from try_parse_tool_calls

'[{\n    "id": "call_dks134",\n    "type": "function",\n    "function": { \n        "name":"update",\n        "arguments": \'{"items": {"name": "test_name"}}\'\n    }\n}]'

Unsuccessful parsing with inclusion of `]` in the arguments property.
```python
bugged_role_prompt = \
"""
## tool_calls: 
[{
    "id": "call_dks134",
    "type": "function",
    "function": { 
        "name":"update",
        "arguments": '{"items": [{"name": "test_name"}]}'
    }
}]
"""
bugged = try_parse_tool_calls(bugged_role_prompt)

Match group from try_parse_tool_calls

'[{\n    "id": "call_dks134",\n    "type": "function",\n    "function": { \n        "name":"update",\n        "arguments": \'{"items": [{"name": "test_name"}]'

The bugged match group ends at the closing of the "items" property rather than the full tool call definition.

Expected behavior
Tool call definitions with arrays in arguments are correctly parsed.

Running Information(please complete the following information):

  • Promptflow Package Version using pf -v: 1.11.0
  • Operating System: MacOS Sonoma 14.5
  • Python Version using python --version: python==3.12.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants