Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

video captioning tool #1593

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Conversation

Ayush0054
Copy link
Contributor

@Ayush0054 Ayush0054 commented Dec 17, 2024

Description

Please include:

  • added video tool : which extracts audio from video , transcribes video , creates srt file of transcription , creates video with captions
  • added example cookbook

Fixes # (issue)

Type of change

Please check the options that are relevant:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Model update
  • Infrastructure change

Checklist

  • My code follows Phidata's style guidelines and best practices
  • I have performed a self-review of my code
  • I have added docstrings and comments for complex logic
  • My changes generate no new warnings or errors
  • I have added cookbook examples for my new addition (if needed)
  • I have updated requirements.txt/pyproject.toml (if needed)
  • I have verified my changes in a clean environment

Additional Notes

Include any deployment notes, performance implications, or other relevant information:

@manthanguptaa manthanguptaa marked this pull request as draft December 18, 2024 08:29
@Ayush0054 Ayush0054 changed the title WIP: video captioning tool video captioning tool Dec 19, 2024
@Ayush0054 Ayush0054 marked this pull request as ready for review December 19, 2024 07:34
phi/tools/video.py Outdated Show resolved Hide resolved
phi/tools/video.py Outdated Show resolved Hide resolved
phi/tools/video.py Outdated Show resolved Hide resolved
video = VideoFileClip(video_path)
video.audio.write_audiofile(output_path)
print(f"Audio extracted to {output_path}")
return output_path
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return the full string that you logged

phi/tools/video.py Outdated Show resolved Hide resolved
cookbook/agents/48_video_caption_agent.py Outdated Show resolved Hide resolved
cookbook/agents/48_video_caption_agent.py Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add setup steps to install ffmpeg

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we not have this video? Let's have some normal video

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on it

Comment on lines 33 to 43
# Create temp directory if it doesn't exist
temp_dir = Path("/tmp/video_captions")
temp_dir.mkdir(parents=True, exist_ok=True)


video_caption_agent.print_response(
"Generate captions for cookbook/examples/caption_video_tool/trump.mp4 and embed them in the video"
)
# video_caption_agent.print_response(
# "read the captions for /Users/ayushjha/Downloads/videoplayback (1).mp4 and summarize them"
# )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The video doesn't get created in the tmp folder but instead in the same examples folder. We also need to move that video from examples folder

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants