Add image support to the OpenAI provider #53
Conversation
…_text() or generate_data() with models that can process images.
WalkthroughThe changes in this pull request involve modifications to the Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (2)
simplemind/providers/openai.py (2)
191-210: Refactor duplicated code for message construction withimage_urlThe logic for constructing the
messageslist and appending theimage_urlwhen provided is duplicated across thestructured_response,generate_text, andgenerate_stream_textmethods. To improve maintainability and reduce code duplication, consider refactoring this code into a helper method.Here's how you could implement the helper method:
def _build_messages(self, prompt: str, image_url: str | None = None): messages = [ { "role": "user", "content": [ { "type": "text", "text": prompt } ] }, ] # Add an image (url or base64-encoded) to the message if provided. if image_url: messages[0]["content"].append( { "type": "image_url", "image_url": { "url": image_url } } ) return messagesThen, update the methods to use this helper:
messages = self._build_messages(prompt, image_url)Also applies to: 225-244, 261-280
202-202: Replace triple-quoted strings with inline commentsThe triple-quoted strings at lines 202, 236, and 272 are being used as comments but are not associated with any function or class as docstrings. It's better to use inline comments with
#to improve code readability and avoid confusion.Apply this diff to replace the triple-quoted strings:
- """Add an image (url or base64-encoded) to the message if provided.""" + # Add an image (url or base64-encoded) to the message if provided.Also applies to: 236-236, 272-272
|
Nice one! It addresses #52 |
|
Instead of tacking this into |
|
It's still generating text as the output, not an image. This is just another input to the text generation process so it can describe an image. |
|
Understood, thanks for the elaboration! |
Add the ability to specify an image_url parameter (accepts url or base64-encoded image) when using the generate_text() or generate_data() functions with the openai provider for use with models that can process images.
Implements the message structure defined in OpenAI's Vision guide: https://platform.openai.com/docs/guides/vision#quickstart
Tested with models on OpenRouter and Groq since they provide OpenAI-compatible APIs too.
Summary by CodeRabbit
New Features
image_urlparameter in key methods.Bug Fixes