> ## Documentation Index
> Fetch the complete documentation index at: https://dsrs.herumbshandilya.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> TypedMetric, Eval, and the trainset evaluation loop

Evaluation is how you tell an optimizer what "good" means. The `evaluate` module provides a metric trait, a result type, a concurrent evaluation loop, and helper functions that build rich textual feedback. A metric sees fully typed data on both sides: the trainset row with all its gold data and the prediction as the module actually produced it.

## Writing and running a metric

A minimal exact-match metric for a QA task, over a [row struct](/docs/components/data) trainset:

```rust theme={null}
use anyhow::Result;
use dspy_rs::{Eval, Example, Predict, Predicted, Trace, TypedMetric};

#[derive(Example, Clone, Debug, serde::Serialize)]
struct QARow {
    question: String,
    answer: String,
}

struct ExactMatchMetric;

impl TypedMetric<QARow, Predict<QA>> for ExactMatchMetric {
    async fn evaluate(
        &self,
        example: &QARow,
        prediction: &Predicted<QAOutput>,
        _trace: Option<&Trace>,
    ) -> Result<Eval> {
        let expected = example.answer.trim().to_lowercase();
        let actual = prediction.answer.trim().to_lowercase();
        Ok(Eval::score((expected == actual) as u8 as f64))
    }
}
```

`evaluate_trainset` runs the module on every example, scores each rollout with the metric, and `average_score` reduces the results to one number:

```rust theme={null}
use dspy_rs::{average_score, evaluate_trainset};

let evals = evaluate_trainset(&module, &trainset, &ExactMatchMetric).await?;
println!("Average: {:.3}", average_score(&evals));
```

## TypedMetric

`TypedMetric<E, M>` is the metric trait. It is generic over `E` (the trainset row — any struct you like, see [Data](/docs/components/data)) and `M: Module`, so the metric sees fully typed data: the row with all its fields, and the prediction as the module actually produced it (for example `WithReasoning<QAOutput>` from `ChainOfThought`). Because the metric receives the row rather than a signature-shaped pair, gold data need not fit the module's output type — a row can carry metric-only fields the module never sees, such as HotpotQA supporting facts.

```rust theme={null}
pub trait TypedMetric<E, M>: Send + Sync
where
    M: Module,
{
    async fn evaluate(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: Option<&Trace>,
    ) -> Result<Eval>;

    // Optional; the default returns no span scores.
    async fn evaluate_spans(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: &Trace,
    ) -> Result<Vec<(SpanId, Eval)>> {
        Ok(Vec::new())
    }
}
```

| Argument     | Meaning                                                                                                                                                                                                                           |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `example`    | The trainset row under evaluation: the full `E`, including gold labels and metric-only fields (the ground truth)                                                                                                                  |
| `prediction` | The module's `Predicted<M::Output>`                                                                                                                                                                                               |
| `trace`      | The rollout's execution [`Trace`](/docs/components/traces) when the caller captured one. The evaluation loop always passes `Some`; direct callers may pass `None`. Slice it per component with `trace.for_component("retriever")` |

Return `Eval::score(f64)` for a numerical score, `Eval::with_feedback(f64, text)` to also explain why. Scores are 0.0 to 1.0 by convention.

## Per-span credit

`evaluate` assigns one score to the whole rollout. For a multi-step module that single score over-credits: a good final answer marks every intermediate `Predict` call as good, including a step a later call had to recover from. `evaluate_spans` is the optional hook for per-span credit. The evaluation loop calls it once per traced rollout, after `evaluate`, and stamps each returned `Eval` onto its span (`Span::eval`); pairs whose id is not in the trace are ignored.

```rust theme={null}
async fn evaluate_spans(
    &self,
    example: &QARow,
    _prediction: &Predicted<QAOutput>,
    trace: &Trace,
) -> Result<Vec<(SpanId, Eval)>> {
    // Score each draft call on its own answer; the refine step may have
    // recovered from a bad one.
    Ok(trace
        .for_component("draft")
        .filter_map(|span| {
            let answer = span.output.as_ref()?.get("answer")?.as_str()?;
            let score = (answer == example.answer) as u8 as f64;
            Some((span.id, Eval::score(score)))
        })
        .collect())
}
```

Demo harvesting (`BootstrapFewShot`, `MIPROv2`, `SIMBA`) prefers a span's own eval over the rollout score when gating and ranking demo candidates, so a scored-down span stays out of the demo pool even when its rollout won, and a scored-up span qualifies even when its rollout lost. Spans you leave out keep whole-rollout credit, and a metric that implements only `evaluate` behaves exactly as before. See [Optimizers](/docs/components/optimizers) for the harvesting semantics.

## Eval and Rollout

`Eval` is defined in the trace module and re-exported by `evaluate`.

| Type      | Definition                                 | Notes                                                                                                                                                         |
| --------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Eval`    | `{ score: f64, feedback: Option<String> }` | Constructors: `Eval::score(score)` and `Eval::with_feedback(score, feedback)`                                                                                 |
| `Rollout` | `(Eval, Trace)`                            | One evaluated rollout: the metric result plus the execution trace that produced it, with `Trace::outcome` filled in (serialized output, the `Eval`, duration) |

The public evaluation entry points return `Vec<Eval>`; the traced `Rollout` path is what optimizers consume internally.

## Evaluation functions

| Item                                 | Signature                                                                                                    | Behavior                                                                                                                                                                                                                                                            |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluate_trainset`                  | `async fn(module: &M, trainset: &[E], metric: &MT) -> Result<Vec<Eval>>` where `E: ToInput<M::Input> + Sync` | Runs the module on every row (projected via `to_input()`) and scores each with the metric. Results come back in trainset order. Any `Module::call` or `TypedMetric::evaluate` failure propagates immediately; for fault-tolerant batching use `forward_all` instead |
| `evaluate_trainset_with_concurrency` | `async fn(module, trainset, metric, max_concurrency: usize) -> Result<Vec<Eval>>`                            | Same loop with an explicit concurrency level. `max_concurrency` LM calls run in flight at once (clamped to at least 1). Use `1` for strictly sequential evaluation on rate-limited providers                                                                        |
| `DEFAULT_EVAL_CONCURRENCY`           | `const usize = 16`                                                                                           | Concurrency used by `evaluate_trainset`                                                                                                                                                                                                                             |
| `average_score`                      | `fn(evals: &[Eval]) -> f64`                                                                                  | Arithmetic mean of scores; returns `0.0` for an empty slice                                                                                                                                                                                                         |

Each example runs inside a trace capture scope and the metric receives that rollout's `Trace`. Metric evaluation itself happens outside the scope, so LM-as-judge metrics do not pollute the execution trace.

## Metrics and optimizers

Optimizers call the evaluation loop internally; the metric you hand them determines what they can do with the results.

| Metric style        | Constructor                      | Sufficient for                                                                                             |
| ------------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Score-only          | `Eval::score(f64)`               | `COPRO`, `MIPROv2`, and any optimizer that ranks candidates by `average_score`                             |
| Score plus feedback | `Eval::with_feedback(f64, text)` | Required by `GEPA`, which feeds the textual feedback into its reflection step to guide evolutionary search |

The feedback helpers above exist mainly to serve GEPA: a metric that explains why a rollout scored low gives the reflection model something concrete to fix. See [Optimizers](/docs/components/optimizers) for the optimizer-side contract.

## See also

* [Optimizers](/docs/components/optimizers)
* [Traces](/docs/components/traces)
* [Data](/docs/components/data)
* [GEPA](/docs/optimizers/gepa)
* [Example: evaluate on HotpotQA](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/03-evaluate-hotpotqa.rs)
* [Example: optimize on HotpotQA](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/04-optimize-hotpotqa.rs)
* [Example: GEPA sentiment](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/09-gepa-sentiment.rs)
* [Example: GEPA LLM judge](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/10-gepa-llm-judge.rs)