Topic-Shift Chunker

A document chunker that encodes sentences and places boundaries where embedding discontinuities indicate topic shifts, yielding variable-size coherent chunks.

Responsibility. Splits documents at detected topic boundaries.

Also known as: Semantic-aware splitting, Embedding-based chunker

Variant of Document Chunker abstract

When to choose. Choose when chunk coherence matters more than uniform chunk size.