Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for base64 encoded audio files #380

Closed
wants to merge 2 commits into from
Closed

feat: add support for base64 encoded audio files #380

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ed872c4
feat: add support for base64 encoded audio files
mdimado Jan 21, 2025
73ddef7
Update src/rai/rai/messages/multimodal.py
mdimado Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions src/rai/rai/messages/multimodal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,17 @@ class MultimodalArtifact(TypedDict):

class MultimodalMessage(BaseMessage):
images: Optional[List[str]] = None
audios: Optional[Any] = None
audios: Optional[List[str]] = None

def __init__(
self,
**kwargs: Any,
):
super().__init__(**kwargs) # type: ignore

if self.audios not in [None, []]:
raise ValueError("Audio is not yet supported")
# remove the audio blocking check
# if self.audios not in [None, []]:
# raise ValueError("Audio is not yet supported")
Comment on lines +38 to +40
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you remove a block of code please don't leave it as a comment. This will help maintain the codebase clean.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my bad, I will not leave it as a comment. I will remove it in the next pr


_content: List[Union[str, Dict[str, Union[Dict[str, str], str]]]] = []

Expand All @@ -56,6 +57,19 @@ def __init__(
for image in self.images
]
_content.extend(_image_content)

# audio content handling (used audio/wav as MIME type)
if isinstance(self.audios, list):
_audio_content = [
{
"type": "audio_url",
"audio_url": {
"url": f"data:audio/wav;base64,{audio}",
},
}
for audio in self.audios
]
_content.extend(_audio_content)
self.content = _content

@property
Expand Down
Loading