Add semantic chunking pipeline

[davidmezzetti]

Evaluate using the Chonkie library to add more sophisticated chunking to txtai.

In reviewing the code, it appears that an interface wrapping a txtai vectors instance could be passed in. With this, any supported txtai vectors model (huggingface, sentence-transformers, LiteLLM, llama-cpp etc) could be used.

[davidmezzetti]

Docling also has some chunkers that are worth considering: https://ds4sd.github.io/docling/examples/hybrid_chunking/

[bhavnicksm]

Hey @davidmezzetti! 👋

I'm glad that you found chonkie useful for txtai! I would be happy to help accelerate the integration of Chonkie here~ Please let me know if there is any support needed.

BTW, we do offer support for customizable embeddings for SemanticChunker and SDPMChunker! Unfortunately, the newly added LateChunker is limited to sentence-transformers only at the moment (but that makes a very strong chunking technique).

Thanks 😊