ollama tool calling with Vercel's AI SDK

[mfshaho]

the AI SDK says that in order to call tools using the LLM, The model needs to support calling tools.

using this Ollama Provider, which ollama models are capable of tool calling?

[sgomez]

Hi, @mfshaho. I will transfer this issue as a discussion. Anyway, the problem is that Ollama API doesn't support function call yet. So right now it is not possible to do it directly.

It is possible to do it with prompt injection and I am doing some progress in that line. I hope to have a way to do it soon.

[lebrunel]

Just a headsup that Ollama now supports tool calling, but it seems only via the OpenAI compatible endpoint. eg, see the mistral-nemo model.

I don't know if the feature will soon follow in on the Ollama-specific chat endpoint, but my hunch is maybe not, feels like the OpenAI compatible endpoint is probably the primary endpoint to use for Ollama moving forward.

[krsilas]

Does it work for you? When I try to do tool calling via the OpenAI api with streaming, I do not get any responses. Prompt injection via the ollama api works though.

[lebrunel]

Yes... well kind of :P

Using ollama 0.2.8 and the new mistral-nemo model.

• without streaming - seems to work as expected
• with streaming - I can define the tools in structured request, but the tool calls are not returned in the structured response, but I can see them in the raw streaming text of the response.

[sgomez]

Thanks @lebrunel

This provider made sense because in Ollama the OpenAI API did not work well. If they finally manage to make Ollama 100% compatible with the OpenAI API, I will close the repository.

Personally I think it is absurd that Ollama has two different APIs and they should focus on being compatible with OpenAI.

[sgomez]

Finally they added support to its own API, I hope to release a new version using it soon.

https://x.com/ollama/status/1816633415770472485