Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Langchain::Assistant when using OpenAI accepts a message with image_url #799

Merged
merged 2 commits into from
Sep 30, 2024

Conversation

andreibondarev
Copy link
Collaborator

@andreibondarev andreibondarev commented Sep 30, 2024

Langchain::Assistant now accepts image_urls when it uses the OpenAI LLM.

Example:

llm = Langchain::LLM::OpenAI.new(api_key: ENV["OPENAI_API_KEY"])

assistant = Langchain::Assistant.new(
  llm: llm,
  instructions: "You're a helpful AI assistant"
)

messages = assistant.add_message_and_run content:"Please describe the image", image_url:"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px
-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

messages.last.content
#=> "The image showcases a tranquil landscape featuring a wooden boardwalk that meanders through a vibrant green field. The boardwalk is flanked by tall grasses and shrubs, leading the viewer's gaze toward the horizon. Above, the sky is a bright blue with soft, wispy clouds scattered throughout, creating a serene atmosphere. The overall scene evokes a sense of peace and invites exploration of the natural surroundings."

@andreibondarev andreibondarev linked an issue Sep 30, 2024 that may be closed by this pull request
6 tasks
@andreibondarev andreibondarev merged commit 33ad323 into main Sep 30, 2024
7 checks passed
@andreibondarev andreibondarev deleted the asssistant-support-images branch September 30, 2024 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ability to send images to the Assistant
1 participant