Evaluation

Evaluation is how you tell an optimizer what “good” means. The evaluate module provides a metric trait, a result type, a concurrent evaluation loop, and helper functions that build rich textual feedback. A metric sees fully typed data on both sides: the trainset row with all its gold data and the prediction as the module actually produced it.

Writing and running a metric

A minimal exact-match metric for a QA task, over a row struct trainset:

use anyhow::Result;
use dspy_rs::{Eval, Example, Predict, Predicted, Trace, TypedMetric};

#[derive(Example, Clone, Debug, serde::Serialize)]
struct QARow {
    question: String,
    answer: String,
}

struct ExactMatchMetric;

impl TypedMetric<QARow, Predict<QA>> for ExactMatchMetric {
    async fn evaluate(
        &self,
        example: &QARow,
        prediction: &Predicted<QAOutput>,
        _trace: Option<&Trace>,
    ) -> Result<Eval> {
        let expected = example.answer.trim().to_lowercase();
        let actual = prediction.answer.trim().to_lowercase();
        Ok(Eval::score((expected == actual) as u8 as f64))
    }
}

evaluate_trainset runs the module on every example, scores each rollout with the metric, and average_score reduces the results to one number:

use dspy_rs::{average_score, evaluate_trainset};

let evals = evaluate_trainset(&module, &trainset, &ExactMatchMetric).await?;
println!("Average: {:.3}", average_score(&evals));

TypedMetric

TypedMetric<E, M> is the metric trait. It is generic over E (the trainset row — any struct you like, see Data) and M: Module, so the metric sees fully typed data: the row with all its fields, and the prediction as the module actually produced it (for example WithReasoning<QAOutput> from ChainOfThought). Because the metric receives the row rather than a signature-shaped pair, gold data need not fit the module’s output type — a row can carry metric-only fields the module never sees, such as HotpotQA supporting facts.

pub trait TypedMetric<E, M>: Send + Sync
where
    M: Module,
{
    async fn evaluate(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: Option<&Trace>,
    ) -> Result<Eval>;

    // Optional; the default returns no span scores.
    async fn evaluate_spans(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: &Trace,
    ) -> Result<Vec<(SpanId, Eval)>> {
        Ok(Vec::new())
    }
}

Argument	Meaning
`example`	The trainset row under evaluation: the full `E`, including gold labels and metric-only fields (the ground truth)
`prediction`	The module’s `Predicted<M::Output>`
`trace`	The rollout’s execution `Trace` when the caller captured one. The evaluation loop always passes `Some`; direct callers may pass `None`. Slice it per component with `trace.for_component("retriever")`

Return Eval::score(f64) for a numerical score, Eval::with_feedback(f64, text) to also explain why. Scores are 0.0 to 1.0 by convention.

Per-span credit

evaluate assigns one score to the whole rollout. For a multi-step module that single score over-credits: a good final answer marks every intermediate Predict call as good, including a step a later call had to recover from. evaluate_spans is the optional hook for per-span credit. The evaluation loop calls it once per traced rollout, after evaluate, and stamps each returned Eval onto its span (Span::eval); pairs whose id is not in the trace are ignored.

async fn evaluate_spans(
    &self,
    example: &QARow,
    _prediction: &Predicted<QAOutput>,
    trace: &Trace,
) -> Result<Vec<(SpanId, Eval)>> {
    // Score each draft call on its own answer; the refine step may have
    // recovered from a bad one.
    Ok(trace
        .for_component("draft")
        .filter_map(|span| {
            let answer = span.output.as_ref()?.get("answer")?.as_str()?;
            let score = (answer == example.answer) as u8 as f64;
            Some((span.id, Eval::score(score)))
        })
        .collect())
}

Demo harvesting (BootstrapFewShot, MIPROv2, SIMBA) prefers a span’s own eval over the rollout score when gating and ranking demo candidates, so a scored-down span stays out of the demo pool even when its rollout won, and a scored-up span qualifies even when its rollout lost. Spans you leave out keep whole-rollout credit, and a metric that implements only evaluate behaves exactly as before. See Optimizers for the harvesting semantics.

Eval and Rollout

Eval is defined in the trace module and re-exported by evaluate.

Type	Definition	Notes
`Eval`	`{ score: f64, feedback: Option<String> }`	Constructors: `Eval::score(score)` and `Eval::with_feedback(score, feedback)`
`Rollout`	`(Eval, Trace)`	One evaluated rollout: the metric result plus the execution trace that produced it, with `Trace::outcome` filled in (serialized output, the `Eval`, duration)

The public evaluation entry points return Vec<Eval>; the traced Rollout path is what optimizers consume internally.

Evaluation functions

Item	Signature	Behavior
`evaluate_trainset`	`async fn(module: &M, trainset: &[E], metric: &MT) -> Result<Vec<Eval>>` where `E: ToInput<M::Input> + Sync`	Runs the module on every row (projected via `to_input()`) and scores each with the metric. Results come back in trainset order. Any `Module::call` or `TypedMetric::evaluate` failure propagates immediately; for fault-tolerant batching use `forward_all` instead
`evaluate_trainset_with_concurrency`	`async fn(module, trainset, metric, max_concurrency: usize) -> Result<Vec<Eval>>`	Same loop with an explicit concurrency level. `max_concurrency` LM calls run in flight at once (clamped to at least 1). Use `1` for strictly sequential evaluation on rate-limited providers
`DEFAULT_EVAL_CONCURRENCY`	`const usize = 16`	Concurrency used by `evaluate_trainset`
`average_score`	`fn(evals: &[Eval]) -> f64`	Arithmetic mean of scores; returns `0.0` for an empty slice

Each example runs inside a trace capture scope and the metric receives that rollout’s Trace. Metric evaluation itself happens outside the scope, so LM-as-judge metrics do not pollute the execution trace.

Metrics and optimizers

Optimizers call the evaluation loop internally; the metric you hand them determines what they can do with the results.

Metric style	Constructor	Sufficient for
Score-only	`Eval::score(f64)`	`COPRO`, `MIPROv2`, and any optimizer that ranks candidates by `average_score`
Score plus feedback	`Eval::with_feedback(f64, text)`	Required by `GEPA`, which feeds the textual feedback into its reflection step to guide evolutionary search

The feedback helpers above exist mainly to serve GEPA: a metric that explains why a rollout scored low gives the reflection model something concrete to fix. See Optimizers for the optimizer-side contract.

Get Started

Building Blocks

Programs as Data

Tools & Agents

Evaluation & Optimization

Utilities

Writing and running a metric

TypedMetric

Per-span credit

Eval and Rollout

Evaluation functions

Metrics and optimizers

See also

​Writing and running a metric

​TypedMetric

​Per-span credit

​Eval and Rollout

​Evaluation functions

​Metrics and optimizers

​See also

Writing and running a metric

TypedMetric

Per-span credit

Eval and Rollout

Evaluation functions

Metrics and optimizers

See also