windlass.providers.evaluation.builtin

builtin

Built-in evaluation metrics.

Two families, deliberately separated:

Lexical metrics need no model and no dependencies. They are fast, free and deterministic, which makes them the right choice for CI regression gates: exact_match, f1, rouge_l, answer_relevancy_lexical, context_precision_lexical, context_recall_lexical.

LLM-judged metrics need a judge model and cost tokens, but measure things lexical overlap cannot: faithfulness (is the answer supported by the retrieved context?), answer_relevancy, answer_correctness, context_relevancy.

Faithfulness is the one to watch. It is the direct measure of hallucination in a RAG system, and it is the metric that catches a retrieval regression before your users do.

Example

from windlass.interfaces.evaluator import EvalSample ev = BuiltinEvaluator(metrics=["exact_match", "f1"]) report = ev.evaluate([EvalSample(question="q", answer="the cat", reference="the cat")]) report.summary["exact_match"], report.summary["f1"] (1.0, 1.0)

BuiltinEvaluator

BuiltinEvaluator(
    *,
    metrics: Sequence[str] | None = None,
    threshold: float = 0.5,
    llm: Any = None,
    concurrency: int | None = None,
    **config: Any
)

Bases: Evaluator

Windlass's own evaluation metrics.

Parameters:

Name	Type	Description	Default
metrics	Sequence[str] | None	Metric names from :data:LEXICAL_METRICS and :data:JUDGED_METRICS. Defaults to the lexical set, so evaluation works with no model configured.	None
threshold	float	Score at or above which a result passes.	0.5
llm	Any	Judge model. Required only for :data:JUDGED_METRICS.	None
concurrency	int | None	Maximum simultaneous sample evaluations.	None
**config	Any	Forwarded to :class:~windlass.interfaces.evaluator.Evaluator.	{}

Raises:

Type	Description
ValueError	For an unknown metric name.
EvaluationError	When a judged metric is requested with no judge model.

Performance

Lexical metrics are pure Python and effectively free. Each judged metric costs one model call per sample, so a 4-metric run over 500 samples is 2,000 calls — batch it and use a small judge model.

Source code in src\windlass\providers\evaluation\builtin.py

def __init__(
    self,
    *,
    metrics: Sequence[str] | None = None,
    threshold: float = 0.5,
    llm: Any = None,
    concurrency: int | None = None,
    **config: Any,
) -> None:
    chosen = list(metrics or self.default_metrics())
    unknown = set(chosen) - set(LEXICAL_METRICS) - set(JUDGED_METRICS)
    if unknown:
        raise ValueError(
            f"Unknown metric(s): {', '.join(sorted(unknown))}. "
            f"Available: {', '.join(sorted(self.available_metrics()))}"
        )
    super().__init__(
        metrics=chosen, threshold=threshold, llm=llm, concurrency=concurrency, **config
    )
    if any(m in JUDGED_METRICS for m in chosen) and llm is None:
        self._require_llm()

default_metrics classmethod

default_metrics() -> tuple[str, ...]

Return the default metric set: lexical only, so no model is needed.

Source code in src\windlass\providers\evaluation\builtin.py

@classmethod
def default_metrics(cls) -> tuple[str, ...]:
    """Return the default metric set: lexical only, so no model is needed."""
    return ("exact_match", "f1", "rouge_l", "answer_relevancy_lexical")

available_metrics classmethod

available_metrics() -> tuple[str, ...]

Return every metric this evaluator supports.

Source code in src\windlass\providers\evaluation\builtin.py

@classmethod
def available_metrics(cls) -> tuple[str, ...]:
    """Return every metric this evaluator supports."""
    return LEXICAL_METRICS + JUDGED_METRICS

aevaluate_sample async

aevaluate_sample(sample: EvalSample) -> list[EvaluationResult]

Score one sample against every configured metric.

Parameters:

Name	Type	Description	Default
sample	EvalSample	The interaction to score.	required

Returns:

Type	Description
list[EvaluationResult]	One result per metric. A metric whose inputs are missing (a
list[EvaluationResult]	reference-based metric with no reference) is skipped rather than
list[EvaluationResult]	scored zero, so averages stay honest.

Source code in src\windlass\providers\evaluation\builtin.py

async def aevaluate_sample(self, sample: EvalSample) -> list[EvaluationResult]:
    """Score one sample against every configured metric.

    Args:
        sample: The interaction to score.

    Returns:
        One result per metric. A metric whose inputs are missing (a
        reference-based metric with no reference) is skipped rather than
        scored zero, so averages stay honest.
    """
    lexical = [m for m in self.metrics if m in LEXICAL_METRICS]
    judged = [m for m in self.metrics if m in JUDGED_METRICS]

    results = [r for r in (self._lexical(name, sample) for name in lexical) if r]
    if judged:
        produced = await gather_bounded(
            [self._judged(name, sample) for name in judged],
            limit=len(judged),
            return_exceptions=True,
        )
        for name, outcome in zip(judged, produced, strict=True):
            if isinstance(outcome, BaseException):
                self._log.warning("Metric %s failed for %s: %s", name, sample.id, outcome)
                continue
            if outcome:
                results.append(outcome)
    return results