What about open source llms?

[FA-UC-HR]

What do you think about using local / self hosted llms? Like ollama with a vision model? (for example like this https://ollama.com/blog/vision-models)

Maybe it's slower, maybe more expensive than a Pro subscription if you factor in your own hardware or electricity costs, but you don't have to upload your terabyte image collection to google/....

But maybe I'm missing something with this idea.

[bmachek]

Hm. definitely worth thinking about it. Don't know much about running local llms, yet.. ;-)
Will give it a try in a few days.. :-)

[bmachek]

I hacked something together yesterday for use with Ollama REST API. But the API is very limited compared to Gemini or ChatGPT. On the other hand the open webui API looks more promising. I will give that a try as well :-)

[bmachek]

I'm stuck at the moment. There's a bug in open web ui, concerning image uploads.....

[FA-UC-HR]

Which version do you use?
When I tried the bundled docker setup (webui + ollama) the combination didn't accept the vision model.
Using newest ollama and webui (installed with pip) image upload works in the webui, but I didn't try the API endpoints by myself.

[bmachek]

This is a pure API problem, using the interactive web UI works just fine. Even with llama3.2 vision. I'm running the latest docker web UI and ollama latest from the downloads.

[bmachek]

Ok I got much further now. Upload has to be base64 encoded.
I've got another problem now, which I'm also trying to address with the dev of open webui.

Stay tuned. :-)

[bmachek]

This is being tracked in #13 now.