windlass.providers.preprocessors.enrich

enrich

OCR fallback and LLM-driven metadata extraction.

Two preprocessors that reach outside the text they are given:

• :class:OCRPreprocessor rescues scanned documents. pypdf returns empty text for an image-only PDF; this notices and re-reads the file with OCR.
• :class:LLMMetadataPreprocessor asks a model for structured metadata — title, summary, topics, entities, document type — which turns a flat corpus into something you can filter and route on.

Example

from windlass.core.types import Document from windlass.providers.llm.fake import FakeLLM llm = FakeLLM(responses=['{"title": "Q3 Report", "topics": ["revenue"]}']) doc = LLMMetadataPreprocessor(llm=llm).process([Document(content="...")])[0] doc.metadata["title"] 'Q3 Report'

OCRPreprocessor

OCRPreprocessor(
    *,
    min_chars: int = 50,
    language: str = "eng",
    dpi: int = 300,
    max_pages: int = 50,
    **config: Any
)

Bases: Preprocessor

OCR fallback for documents whose text layer is missing or unusable.

A scanned PDF is a stack of images. Text extraction returns nothing, and the document silently disappears from your index. This preprocessor detects that and re-reads the source with OCR.

Parameters:

Name	Type	Description	Default
min_chars	int	Documents with at least this much text are passed through untouched — OCR is only a fallback, never a replacement.	50
language	str	Tesseract language code(s).	'eng'
dpi	int	Rendering resolution for PDF pages. 300 is the usual sweet spot; higher is slower with diminishing returns.	300
max_pages	int	Ceiling on pages rendered per PDF, to bound the cost of a mistakenly-included 900-page scan.	50
**config	Any	Forwarded to :class:~windlass.interfaces.preprocessor.Preprocessor.	{}

Raises:

Type	Description
MissingDependencyError	When the OCR extras are not installed and a document actually needs OCR.

Note

PDF rendering needs pdf2image plus Poppler, which Windlass does not install for you. When rendering is unavailable the document is passed through unchanged and a warning is logged, rather than failing the run.

Source code in src\windlass\providers\preprocessors\enrich.py

def __init__(
    self,
    *,
    min_chars: int = 50,
    language: str = "eng",
    dpi: int = 300,
    max_pages: int = 50,
    **config: Any,
) -> None:
    super().__init__(**config)
    self.min_chars = min_chars
    self.language = language
    self.dpi = dpi
    self.max_pages = max_pages

aprocess_one async

aprocess_one(document: Document) -> list[Document]

Re-read the document with OCR when its text is missing.

Parameters:

Name	Type	Description	Default
document	Document	The document to check.	required

Returns:

Type	Description
list[Document]	The original document, or an OCR'd replacement.

Source code in src\windlass\providers\preprocessors\enrich.py

async def aprocess_one(self, document: Document) -> list[Document]:
    """Re-read the document with OCR when its text is missing.

    Args:
        document: The document to check.

    Returns:
        The original document, or an OCR'd replacement.
    """
    if len(document.content.strip()) >= self.min_chars:
        return [document]

    source = document.source
    if not source or Path(source).suffix.lower() not in _OCR_SOURCES:
        return [document]

    suffix = Path(source).suffix.lower()
    try:
        text = (
            await self._ocr_pdf(source, document)
            if suffix == ".pdf"
            else await self._ocr_image(source)
        )
    except Exception as exc:
        self._log.warning("OCR fallback failed for %s: %s", source, exc)
        return [document]

    if len(text.strip()) < self.min_chars:
        return [document]

    return [
        document.model_copy(
            update={
                "content": text,
                "metadata": {**document.metadata, "ocr": True, "ocr_language": self.language},
            }
        )
    ]

LLMMetadataPreprocessor

LLMMetadataPreprocessor(
    *,
    llm: LLM | None = None,
    max_input_tokens: int = 1500,
    fields: tuple[str, ...] = (
        "title",
        "summary",
        "topics",
        "entities",
        "document_type",
    ),
    overwrite: bool = False,
    **config: Any
)

Bases: Preprocessor

Extracts structured metadata using a language model.

Parameters:

Name	Type	Description	Default
llm	LLM | None	The model to ask. Use a small, cheap one — this is extraction, not reasoning.	None
max_input_tokens	int	How much of each document to show the model. The opening of a document usually carries its title and topic.	1500
fields	tuple[str, ...]	Metadata keys to keep from the model's reply.	('title', 'summary', 'topics', 'entities', 'document_type')
overwrite	bool	Replace metadata keys that already exist. Off by default, so metadata the loader extracted from the file itself wins.	False
**config	Any	Forwarded to :class:~windlass.interfaces.preprocessor.Preprocessor.	{}

Raises:

Type	Description
ConfigurationError	When no model is supplied.

Performance

One model call per document, run with the batch concurrency of the preprocessor base class. On a large corpus this is the most expensive preprocessor by a wide margin — measure before enabling it on millions of documents.

Source code in src\windlass\providers\preprocessors\enrich.py

def __init__(
    self,
    *,
    llm: LLM | None = None,
    max_input_tokens: int = 1500,
    fields: tuple[str, ...] = ("title", "summary", "topics", "entities", "document_type"),
    overwrite: bool = False,
    **config: Any,
) -> None:
    if llm is None:
        raise ConfigurationError(
            "Metadata extraction needs a language model.",
            hint="Pass llm=Windlass.llm('openai', model='gpt-4o-mini').",
        )
    super().__init__(**config)
    self.llm = llm
    self.max_input_tokens = max_input_tokens
    self.fields = tuple(fields)
    self.overwrite = overwrite

aprocess_one async

aprocess_one(document: Document) -> list[Document]

Ask the model for metadata and merge it in.

A failed or unparseable reply leaves the document untouched — metadata enrichment is an improvement, never a gate.

Parameters:

Name	Type	Description	Default
document	Document	The document to enrich.	required

Returns:

Type	Description
list[Document]	A single-element list.

Source code in src\windlass\providers\preprocessors\enrich.py

async def aprocess_one(self, document: Document) -> list[Document]:
    """Ask the model for metadata and merge it in.

    A failed or unparseable reply leaves the document untouched — metadata
    enrichment is an improvement, never a gate.

    Args:
        document: The document to enrich.

    Returns:
        A single-element list.
    """
    excerpt = truncate_tokens(document.content, self.max_input_tokens, suffix="\n[…]")
    try:
        completion = await self.llm.acomplete(
            METADATA_PROMPT.format(content=excerpt), max_tokens=400
        )
        extracted = _parse_json(completion.content)
    except Exception as exc:
        self._log.warning("Metadata extraction failed for %s: %s", document.id, exc)
        return [document]

    if not extracted:
        return [document]

    metadata = dict(document.metadata)
    for key in self.fields:
        value = extracted.get(key)
        if value in (None, "", [], {}):
            continue
        if key in metadata and not self.overwrite:
            continue
        metadata[key] = value
    metadata["llm_metadata"] = True
    return [document.model_copy(update={"metadata": metadata})]