New Feature - Text based video search using AI

[arisha07]

Hello,
Thank you for this incredible project!
At Intel, we’ve developed a feature to bring video retrieval support to Shotcut. This allows users to effortlessly search for videos on their filesystem or locate specific events within lengthy videos using natural language. The entire process runs locally, ensuring the user’s video library remains private and secure.
This feature has been optimized using the OpenVINO framework, so it is optimized mainly for Intel hardware. If you’re interested in incorporating this functionality into the Shotcut project, we’d be thrilled to collaborate with you.
Here’s a link to a short video demonstrating the feature:

video-demo.zip

[ddennedy]

Thanks, OpenVINO is interesting, as I have explored it previously. I tried to build it as backend for the Whisper.cpp that we added. However, it was very heavy to compile with many dependencies, some that failed and required changes, and very slow to compile. I am very sensitive to adding many more heavy dependencies because it is primarily my responsibly, there are several targets (CPU arch, OS, package format). Dependencies make builds more brittle, adds maintenance overhead (some changes require changes), and increase download and install size. So, we are discussing an extension system. The next version of Shotcut has a Files view that is a file system browser, and it has a search for the current folder. This would be a good place to add a search option across files. I think the way we would to integrate a feature like this is as a command line program that Shotcut will find and run as a child. For example, typical command line options with newline delimited list of files or JSON output (file path plus time values).
Next week I am traveling for a month and will think about it more.

[arisha07]

Thank you for your quick response! I’d like to share a bit more about how this works:
We’ve implemented a Qt-based client on the Shotcut side that connects to a video retrieval server. This server operates independently from the Shotcut project and can run either locally on the user’s system or the cloud depending on the use-case. The server can be setup via this link https://github.com/intel/openvino-ai-video-retrieval-analysis.git

This architecture helps keep the dependencies needed for Shotcut lightweight and efficient. In other words, OpenVINO isn't needed in Shotcut build to support this feature.

[ddennedy]

I have been thinking about this, and I want to offer something like this to Shotcut users; however:

1. I do not want to spend too much time on it, and I do not want the UI to work like your video (still a good demo).
2. In order for it to be really usable for anyone less than a true geek, the server should be a more self-contained single docker image that has a REST API instead of the complex setup in your linked README.
3. I am soon going to add a metadata database to Shotcut (probably CLucene) that will be indexing visited files and folders. This will collect hard embedded and user-entered metadata that needs to be searchable and not just collected by AI.
4. This indexer will be able to read sidecar metadata files in a few formats including a de facto standard chapters.txt format similar to what you can enter into a YouTube description to provide chapters. (Shotcut can already export timeline markers as a chapters.txt file.)
5. It would be nice if your server can write these in addition to or as an alternative to storing in VDMS.
6. Then, Shotcut would index these files and associate them with the media in its search DB.

I had a chat with Google about the state of chapters sidecar files:

Does VLC support a chapters sidecar file?

While VLC can handle chapters embedded within media files (like MKV), it doesn't natively support reading chapter information from a separate sidecar file.

Here's why and what you can do:

• VLC's Chapter Handling: VLC primarily relies on chapter information that's contained within the media file itself. Formats like MKV have good support for embedded chapters.
• Sidecar Files: Sidecar files are separate files that contain metadata about a media file, such as chapter information. While some media players support them, VLC doesn't have this feature built-in.

Workarounds:

• Embed Chapters: The most reliable way to have chapters in VLC is to embed them directly into the media file. You can use tools like FFmpeg to do this, often reading chapter information from a text-based sidecar file.
• Alternative Players: Some media players, like MPV, do support chapter sidecar files. If this is a crucial feature for you, you might consider trying a different player.

In Summary:

VLC doesn't directly support chapter sidecar files. You'll need to embed the chapter information into the media file itself for VLC to recognize it.

What is the file name convention for mpv to read a chapters sidecar file and what is its format?

MPV is quite flexible with chapter sidecar files, but here's the breakdown of how it works:

File Name Convention:

• Automatic Loading: MPV will automatically try to load a chapter file if it has the same base name as the video file but with a .chapters.txt extension.
  • For example: video.mkv will look for video.chapters.txt
• Explicit Loading: You can also explicitly specify the chapter file using the --chapters-file command-line option when running MPV.

File Format:

MPV supports a simple and human-readable chapter file format:

• One chapter per line: Each line in the file represents a chapter.
• Format: HH:MM:SS Chapter Title
  • HH is the hour (00-99)
  • MM is the minute (00-59)
  • SS is the second (00-59)
  • Chapter Title is the name of the chapter.

Example:

00:00:00 Introduction
00:05:30 First Act
00:12:45 The Plot Thickens
00:20:10 Climax
00:25:50 Resolution

Important Notes:

• UTF-8 Encoding: The chapter file should be saved with UTF-8 encoding to ensure proper display of chapter titles with special characters.
• No Strict Standard: While this format is widely used, there isn't a strict standard for chapter sidecar files. MPV's approach is simple and effective.
• Lua Script: For more advanced features or different chapter file formats, you might find Lua scripts for MPV that extend its capabilities.

By following these conventions, you can easily create chapter sidecar files for your videos and enjoy chapter navigation in MPV.

What other media players support a chapters sidecar?

While MPV is known for its flexibility and support for chapter sidecar files, it's not the only media player that offers this feature. Here are some other media players that support chapter sidecar files:

• IINA: This is a modern media player for macOS that supports a variety of formats and features, including chapter sidecar files. It's known for its user-friendly interface and performance.
• PotPlayer: A popular media player for Windows, PotPlayer also supports chapter sidecar files. It offers a wide range of customization options and features.
• MPC-HC (Media Player Classic - Home Cinema) and MPC-BE (Media Player Classic - Black Edition): These are lightweight media players for Windows that are known for their simplicity and performance. Both support chapter sidecar files.

Important Notes:

• Format Compatibility: The specific chapter file format supported may vary slightly between players. It's always a good idea to check the documentation or settings of the player you're using.
• Automatic Loading: Some players might automatically load chapter sidecar files if they follow a specific naming convention (like MPV's .chapters.txt), while others might require you to manually load the file.

If you're looking for a media player that supports chapter sidecar files, these are some good options to consider. Each player has its own strengths and features, so you can choose the one that best suits your needs and preferences.

---

• If the $(basename).chapters.txt file aready exist, the tool can write an alternative name TBD.
• There are other popular mpv frontends: mpv.net on Windows, and Celluloid on Linux.

[ddennedy]

Given what I wrote above about chapters.txt, it is just an idea, and I do not know how well it works with your tool. Alternatively, I think it would be nice if a user can simply install Docker if needed, run the docker image, and Shotcut's metadata search functions will call your server if found and collate the results with its own.