Manual creation of functions

[GuerrillaCoder]

I have just updated from latest code and got a breaking change for tools that I can't quite see how to get around.

In my app at beginning of each request I read users account and dynamically generate a function list that lets the user interface based on their own configurations. This now throws an error due to internal private static dictionary every time a function is created with same name across all threads.

Looking at the example in the docs it seems now you pass a delegate and the tool call is handled internally but this will not work for me as the only way I have got my application working well is by adding in descriptions for every parameter, not just the method. They have to be tweaked based on individual users data.

In my app I manually build the list of functions but it seems this internal dictionary now makes that approach impossible as there is no method to access this dictionary externally. Maybe there is still a way to do that but I can't see any example of how it is supposed to work.

How should I approach use case where each request has custom functions and multiple concurrent requests?

[StephenHodgson]

This now throws an error due to internal private static dictionary every time a function is created with same name across all threads.

You should only be creating this list one time, should be simple enough to change your list to readonly property or field in your class.

You should only register these once, and can be cleared if needed using the Tool class methods.

Tool.GetAllAvailableTools();
Tool.GetOrCreateTool(instance, "tool-name");
Tool.TryGetTool("tool-name", instance, out var tool);
Tool.IsToolRegistered(tool);
Tool.ClearRegisteredTools();

it seems now you pass a delegate and the tool call is handled internally but this will not work for me as the only way I have got my application working well is by adding in descriptions for every parameter, not just the method

You should be able to do either. I did not change the default behaviour, and you can still specify functions in json form as you did before.

How should I approach use case where each request has custom functions and multiple concurrent requests?

I'd have to see an example, but you should be able to register the tool/function one time at the start of your app, then just use that same list of tools for the chat/assistant requests. Check the test suite or my Unity demo scene for examples.

[GuerrillaCoder]

The test helped but I think maybe my use case still doesn't fit the changes.

I have an AI voice assistant that is powered by our software. People can make forms in a form-builder and then the AI will fill them out. To achieve this when a call connects I read the API and then dynamically generate a tool that matches their form fields and descriptions.

Users need to be able to edit the form and then the next call that connects has the updated fields available in the tool. Clearing all tools would cause all current calls in progress to stop working.

Maybe I am misunderstanding but the new updates seem to make it so you can only define a tool once and then cannot update it. So I would have to make a new tool with unique tool key every time a user updates a setting.

What is the benefit of making the tools a static global list? Is there a way to edit a function that I am not seeing?

[StephenHodgson]

Ok I will try to resolve this in next release. Tracking this along with #299

[GuerrillaCoder]

I think the concept of a mandatory static list of tools isn't going to work. I want my users to be able to define their own tools and callbacks. Having to manage a static list of every tool in memory and sync it with user settings is going to be an unnecessary amount of work. It makes much more sense to just generate this on each request from the DB but the new changes seem to make this impossible.

[StephenHodgson]

You can define your own tools and callbacks. The list isn't meant for you or your users to manage on your own.

I'll sort it out eventually but it is there for a reason.

[sibbl]

I think the concept of a mandatory static list of tools isn't going to work.

I have the exact same feeling. Of course it's easier for most of the use cases for tool/function calling, but I think developers should have control over what's cached where and a static list should not be collected anywhere without the knowledge of the developer and without a way to disable it.

[StephenHodgson]

I think developers should have control over what's cached

I agree and there are ways to manage this cache, I just don't like the idea that the whole cache can be stomped and replaced.
It could lead to memory leaks, dangling delegates and other problems.

There are convenience methods already in the Tool class to help devs manage the cache:

Tool.GetAllAvailableTools();
Tool.GetOrCreateTool(instance, "tool-name");
Tool.TryGetTool("tool-name", instance, out var tool);
Tool.IsToolRegistered(tool);
Tool.ClearRegisteredTools();

Also see my response to the same exact question here:
#303 (reply in thread)

Mostly because I'm concerned about dangling pointers to functions and memory leaks. I think the library should not expose the internal workings of the cache, as a means to protect the api consumer from having to manage this on their own. Besides the library is the one creating and managing those instances internally anyway, so exposing it doesn't make much sense.

[EvoTechMike]

How do we opt out of the cache or bypass it?

[StephenHodgson]

In 8.0.0 you can bypass by using plain json formatted functions. Only tools with instanced callbacks are cached.