Support LocalAI

[dxcore35]

Please, please!!!

Support LocalAI https://www.youtube.com/watch?v=3yPVDI8wZcI
It is already implemented in the plugin https://github.com/logancyang/obsidian-copilot

I don't think it will be too much code to copy relevant code from this project and implement fully local AI.

Thank you very much for your work!

[creuzerm]

This could be as simple as allowing us to change the URL for the ChatGPT endpoint as some of our local libraries now are chatGPT API compatible.

[AndyDavis8]

It would be great if this project support local AI such as Ollama, LM Studio...

[HyeonseoNam]

Hi @dxcore35, @creuzerm , @AndyDavis8 ,
(thanks to @rtuszik), you can now use the OpenAI-compatible LLM API (possibly enabling it for Ollama) on version 1.2.0.
Consider updating the base URL to point to the address that calls your local LLM.

[dxcore35]

Not working!

There is no API key for Ollama.

[rtuszik]

Not working!

There is no API key for Ollama.

I opened #30 for this. Will look into it.

[rtuszik]

@dxcore35
You can just use any random input for the API key. You need to use the following baseurl: http://localhost:11434/v1

[dxcore35]

✅ The base url set
✅ Random api
✅ Connection to Ollama works
🛑 After pressing classify content:

[rtuszik]

From what I can see, The issue here is that the models don't adhere to the format.
Using ollama3.2, I have gotten these sort of responses:

"content": "JSON {\n \"reliability\": 0.9,\n \"outputs\": [\n \"nato\",\n \"treaty\",\n \"international-law\",\n \"war\",\n \"military\",\n \"politics\"\n ]\n}"

where it just prefaces the JSON with JSON.

"content": "JSON output:\n{\n \"reliability\": 0.9,\n \"outputs\": [\n \"international-law\",\n \"treaty\",\n \"nato\",\n \"united-nations\"\n ]\n}\n\nNote: The reliability score is 0.9 because the content appears to be a North Atlantic Treaty Organization (NATO) treaty, which is an international organization and a type of treaty. The other options are not directly related to the content."

where it appends text to the JSON.

Its impossible to account for that lack of prompt adherence. I am not sure if the format parameter works with the OpenAI-node library.

For now I can only recommend to try using different models to see if one will do. Feel free to also send some debug information about the completions you received.

[dxcore35]

I think he need to implement this for Ollama:
https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#json-schema-model

They also implement the structure output few days back! It will force any model to adhere to JSON
https://ollama.com/blog/structured-outputs

[rtuszik]

@dxcore35
it best to edit your comments instead of re-commenting after such short time. I would ask you to follow #31 for updates on this, as this issue is closed.

The format parameter I had mentioned is what you suggest with the structured outputs.

I will be looking into this, though this might require some significant work and possible an alternative to OpenAI-node.

[creuzerm]

Thank you. I can confirm that this works with a remote server, custom model.

My ollama instance does have some environment variables that enable remote serving.

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_ORIGINS=*"