windlass.providers.chunkers.structural

structural

Structure-aware chunking for Markdown and source code.

Generic splitters treat a document as a flat string. These two respect the structure that is already there:

• :class:MarkdownChunker splits on headings and prepends the heading path to every chunk, so a chunk that says "the timeout defaults to 30s" retrieves as "Configuration > Networking > the timeout defaults to 30s". That context is frequently the difference between a hit and a miss.
• :class:CodeChunker splits at function and class boundaries for a dozen languages, so a retrieved chunk is a complete callable rather than its middle eight lines.

Neither has any dependencies.

Example

md = "# Guide\n\nIntro text.\n\n## Setup\n\nRun the installer." chunks = MarkdownChunker(chunk_size=200).split_text(md) "Guide > Setup" in chunks[-1] True

MarkdownChunker

MarkdownChunker(
    *,
    chunk_size: int = 1500,
    overlap: int | None = None,
    max_heading_level: int = 3,
    include_heading_path: bool = True,
    strip_code_fences: bool = True,
    **config: Any
)

Bases: Chunker

Heading-aware Markdown splitter.

Parameters:

Name	Type	Description	Default
chunk_size	int	Target chunk length in characters. Sections longer than this are split further with the recursive splitter.	1500
overlap	int | None	Characters repeated when a section must be sub-split.	None
max_heading_level	int	Deepest heading treated as a split point. 2 splits on # and ## only, which keeps subsections together.	3
include_heading_path	bool	Prepend the breadcrumb (Guide > Setup) to each chunk. Strongly recommended — it is nearly free context.	True
strip_code_fences	bool	Keep fenced code blocks intact rather than letting a # comment inside them be mistaken for a heading.	True
**config	Any	Forwarded to :class:~windlass.interfaces.chunker.Chunker.	{}

Example

chunker = MarkdownChunker(chunk_size=500, include_heading_path=False) chunker.split_text("# A\n\ntext")[0] '# A\n\ntext'

Source code in src\windlass\providers\chunkers\structural.py

def __init__(
    self,
    *,
    chunk_size: int = 1500,
    overlap: int | None = None,
    max_heading_level: int = 3,
    include_heading_path: bool = True,
    strip_code_fences: bool = True,
    **config: Any,
) -> None:
    super().__init__(chunk_size=chunk_size, overlap=overlap, **config)
    self.max_heading_level = max_heading_level
    self.include_heading_path = include_heading_path
    self.strip_code_fences = strip_code_fences
    self._fallback = RecursiveChunker(chunk_size=chunk_size, overlap=overlap, min_chunk_size=0)

split_text

split_text(text: str) -> list[str]

Split Markdown into heading-scoped chunks.

Parameters:

Name	Type	Description	Default
text	str	The Markdown source.	required

Returns:

Type	Description
list[str]	Chunk strings, each prefixed with its heading path when
list[str]	attr:include_heading_path is on.

Source code in src\windlass\providers\chunkers\structural.py

def split_text(self, text: str) -> list[str]:
    """Split Markdown into heading-scoped chunks.

    Args:
        text: The Markdown source.

    Returns:
        Chunk strings, each prefixed with its heading path when
        :attr:`include_heading_path` is on.
    """
    if not text.strip():
        return []

    sections = self._sections(text)
    chunks: list[str] = []
    for path, raw in sections:
        body = raw.strip()
        if not body:
            continue

        prefix = ""
        if path and self.include_heading_path:
            prefix = f"{' > '.join(path)}\n\n"
            # The breadcrumb already names this section, so repeating its
            # own heading line in the body is pure duplication.
            body = _drop_leading_heading(body)
            if not body:
                # A heading with no content of its own contributes to the
                # path of the sections below it and nothing else.
                continue

        whole = f"{prefix}{body}"
        if len(whole) <= self.chunk_size:
            chunks.append(whole)
        else:
            for piece in self._fallback.split_text(body):
                chunks.append(f"{prefix}{piece}")
    return self._merge_undersized(chunks)

CodeChunker

CodeChunker(
    *,
    language: str = "generic",
    chunk_size: int = 800,
    overlap: int | None = None,
    **config: Any
)

Bases: RecursiveChunker

Language-aware source-code splitter.

Parameters:

Name	Type	Description	Default
language	str	One of the keys in :data:LANGUAGE_SEPARATORS, or "generic". Ignored when :meth:for_path picks one for you.	'generic'
chunk_size	int	Target chunk length in characters. Code is dense, so a smaller value than for prose usually works better.	800
overlap	int | None	Characters repeated between chunks. Small values are right for code — duplicating half a function helps nobody.	None
**config	Any	Forwarded to :class:RecursiveChunker.	{}

Raises:

Type	Description
ValueError	For an unknown language.

Example

code = "def a():\n return 1\n\ndef b():\n return 2\n" chunks = CodeChunker(language="python", chunk_size=25).split_text(code) len(chunks) >= 1 True

Source code in src\windlass\providers\chunkers\structural.py

def __init__(
    self,
    *,
    language: str = "generic",
    chunk_size: int = 800,
    overlap: int | None = None,
    **config: Any,
) -> None:
    key = language.lower()
    if key not in LANGUAGE_SEPARATORS:
        raise ValueError(
            f"Unknown language {language!r}. "
            f"Supported: {', '.join(sorted(LANGUAGE_SEPARATORS))}."
        )
    super().__init__(
        chunk_size=chunk_size,
        overlap=overlap,
        separators=LANGUAGE_SEPARATORS[key],
        **config,
    )
    self.language = key

for_path classmethod

for_path(path: str, **config: Any) -> CodeChunker

Build a chunker configured for a file's language.

Parameters:

Name	Type	Description	Default
path	str	File path or name; only the extension is inspected.	required
**config	Any	Forwarded to the constructor.	{}

Returns:

Type	Description
CodeChunker	A chunker using that language's separators, falling back to
CodeChunker	generic for unknown extensions.

Example

CodeChunker.for_path("app/main.py").language 'python' CodeChunker.for_path("notes.xyz").language 'generic'

Source code in src\windlass\providers\chunkers\structural.py

@classmethod
def for_path(cls, path: str, **config: Any) -> CodeChunker:
    """Build a chunker configured for a file's language.

    Args:
        path: File path or name; only the extension is inspected.
        **config: Forwarded to the constructor.

    Returns:
        A chunker using that language's separators, falling back to
        ``generic`` for unknown extensions.

    Example:
        >>> CodeChunker.for_path("app/main.py").language
        'python'
        >>> CodeChunker.for_path("notes.xyz").language
        'generic'
    """
    from pathlib import Path

    suffix = Path(path).suffix.lower()
    return cls(language=_EXTENSION_LANGUAGES.get(suffix, "generic"), **config)