windlass.providers.chunkers.recursive

recursive

Recursive and token-based chunking.

RecursiveChunker is the right default for prose. It splits on the largest natural boundary that produces small-enough pieces — paragraphs first, then lines, sentences, clauses, words, and finally characters — so chunks end where a human would end them instead of mid-word.

TokenChunker does the same thing but measures in model tokens, which is what you want when you must guarantee that k chunks fit inside a context window.

Neither has any dependencies.

Example

text = "First para.\n\nSecond para is longer and keeps going for a while." chunks = RecursiveChunker(chunk_size=40, overlap=5).split_text(text) all(len(c) <= 60 for c in chunks) True

RecursiveChunker

RecursiveChunker(
    *,
    chunk_size: int = 1000,
    overlap: int | None = None,
    separators: tuple[str, ...] | list[str] | None = None,
    **config: Any
)

Bases: Chunker

Character-based recursive splitter.

Parameters:

Name	Type	Description	Default
chunk_size	int	Target chunk length in characters.	1000
overlap	int | None	Characters repeated between consecutive chunks.	None
separators	tuple[str, ...] | list[str] | None	Boundary strings tried in order. The default walks from paragraphs down to individual characters.	None
**config	Any	Forwarded to :class:~windlass.interfaces.chunker.Chunker — notably min_chunk_size, below which a fragment is merged into its neighbour rather than emitted on its own.	{}

Performance

Linear in the length of the text. A 10 MB document splits in well under a second.

Example

chunker = RecursiveChunker(chunk_size=30, overlap=0) parts = chunker.split_text("alpha beta gamma delta epsilon zeta eta") len(parts) > 1 True

Source code in src\windlass\providers\chunkers\recursive.py

def __init__(
    self,
    *,
    chunk_size: int = 1000,
    overlap: int | None = None,
    separators: tuple[str, ...] | list[str] | None = None,
    **config: Any,
) -> None:
    super().__init__(chunk_size=chunk_size, overlap=overlap, **config)
    self.separators = tuple(separators) if separators else DEFAULT_SEPARATORS

split_text

split_text(text: str) -> list[str]

Split text recursively.

Parameters:

Name	Type	Description	Default
text	str	The text to split.	required

Returns:

Type	Description
list[str]	Chunk strings, each at most chunk_size in this chunker's unit
list[str]	(except where a single indivisible token exceeds it).

Source code in src\windlass\providers\chunkers\recursive.py

def split_text(self, text: str) -> list[str]:
    """Split ``text`` recursively.

    Args:
        text: The text to split.

    Returns:
        Chunk strings, each at most ``chunk_size`` in this chunker's unit
        (except where a single indivisible token exceeds it).
    """
    if not text or not text.strip():
        return []
    if self._size(text) <= self.chunk_size:
        return [text.strip()]
    pieces = self._split(text, list(self.separators))
    return self._pack(pieces)

TokenChunker

TokenChunker(
    *,
    chunk_size: int = 512,
    overlap: int | None = None,
    model: str = "gpt-4o",
    **config: Any
)

Bases: RecursiveChunker

Recursive splitter that measures chunks in tokens.

Use this when a hard context budget matters: chunk_size=512 with top_k=8 guarantees roughly 4k tokens of context regardless of how verbose the source text is.

Parameters:

Name	Type	Description	Default
chunk_size	int	Target chunk length in tokens.	512
overlap	int | None	Tokens repeated between consecutive chunks.	None
model	str	Model name used to select the tokeniser.	'gpt-4o'
**config	Any	Forwarded to :class:RecursiveChunker.	{}

Note

Exact token counts require tiktoken (installed by windlass[rag]). Without it, Windlass falls back to a calibrated character-ratio estimate, which is close enough for sizing but not for hard limits.

Example

chunks = TokenChunker(chunk_size=8, overlap=2).split_text( ... "one two three four five six seven eight nine ten eleven twelve" ... ) len(chunks) >= 1 True

Source code in src\windlass\providers\chunkers\recursive.py

def __init__(
    self,
    *,
    chunk_size: int = 512,
    overlap: int | None = None,
    model: str = "gpt-4o",
    **config: Any,
) -> None:
    super().__init__(chunk_size=chunk_size, overlap=overlap, model=model, **config)
    self.model = model