> ## Documentation Index
> Fetch the complete documentation index at: https://dsrs.herumbshandilya.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> TypedMetric, Eval, the trainset evaluation loop, and feedback helper functions

Evaluation is how you tell an optimizer what "good" means. The `evaluate` module provides a metric trait, a result type, a concurrent evaluation loop, and helper functions that build rich textual feedback. A metric sees fully typed data on both sides: the trainset row with all its gold data and the prediction as the module actually produced it.

## Writing and running a metric

A minimal exact-match metric for a QA task, over a [row struct](/docs/components/data) trainset:

```rust theme={null}
use anyhow::Result;
use dspy_rs::{Eval, Example, Predict, Predicted, Trace, TypedMetric};

#[derive(Example, Clone, Debug, serde::Serialize)]
struct QARow {
    question: String,
    answer: String,
}

struct ExactMatchMetric;

impl TypedMetric<QARow, Predict<QA>> for ExactMatchMetric {
    async fn evaluate(
        &self,
        example: &QARow,
        prediction: &Predicted<QAOutput>,
        _trace: Option<&Trace>,
    ) -> Result<Eval> {
        let expected = example.answer.trim().to_lowercase();
        let actual = prediction.answer.trim().to_lowercase();
        Ok(Eval::score((expected == actual) as u8 as f64))
    }
}
```

`evaluate_trainset` runs the module on every example, scores each rollout with the metric, and `average_score` reduces the results to one number:

```rust theme={null}
use dspy_rs::{average_score, evaluate_trainset};

let evals = evaluate_trainset(&module, &trainset, &ExactMatchMetric).await?;
println!("Average: {:.3}", average_score(&evals));
```

## TypedMetric

`TypedMetric<E, M>` is the metric trait. It is generic over `E` (the trainset row — any struct you like, see [Data](/docs/components/data)) and `M: Module`, so the metric sees fully typed data: the row with all its fields, and the prediction as the module actually produced it (for example `WithReasoning<QAOutput>` from `ChainOfThought`). Because the metric receives the row rather than a signature-shaped pair, gold data need not fit the module's output type — a row can carry metric-only fields the module never sees, such as HotpotQA supporting facts.

```rust theme={null}
pub trait TypedMetric<E, M>: Send + Sync
where
    M: Module,
{
    async fn evaluate(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: Option<&Trace>,
    ) -> Result<Eval>;
}
```

| Argument     | Meaning                                                                                                                                                                                                                           |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `example`    | The trainset row under evaluation: the full `E`, including gold labels and metric-only fields (the ground truth)                                                                                                                  |
| `prediction` | The module's `Predicted<M::Output>`                                                                                                                                                                                               |
| `trace`      | The rollout's execution [`Trace`](/docs/components/traces) when the caller captured one. The evaluation loop always passes `Some`; direct callers may pass `None`. Slice it per component with `trace.for_component("retriever")` |

Return `Eval::score(f64)` for a numerical score, `Eval::with_feedback(f64, text)` to also explain why. Scores are 0.0 to 1.0 by convention.

## Eval and Rollout

`Eval` is defined in the trace module and re-exported by `evaluate`.

| Type      | Definition                                 | Notes                                                                                                                                                         |
| --------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Eval`    | `{ score: f64, feedback: Option<String> }` | Constructors: `Eval::score(score)` and `Eval::with_feedback(score, feedback)`                                                                                 |
| `Rollout` | `(Eval, Trace)`                            | One evaluated rollout: the metric result plus the execution trace that produced it, with `Trace::outcome` filled in (serialized output, the `Eval`, duration) |

The public evaluation entry points return `Vec<Eval>`; the traced `Rollout` path is what optimizers consume internally.

## Evaluation functions

| Item                                 | Signature                                                                                                    | Behavior                                                                                                                                                                                                                                                            |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluate_trainset`                  | `async fn(module: &M, trainset: &[E], metric: &MT) -> Result<Vec<Eval>>` where `E: ToInput<M::Input> + Sync` | Runs the module on every row (projected via `to_input()`) and scores each with the metric. Results come back in trainset order. Any `Module::call` or `TypedMetric::evaluate` failure propagates immediately; for fault-tolerant batching use `forward_all` instead |
| `evaluate_trainset_with_concurrency` | `async fn(module, trainset, metric, max_concurrency: usize) -> Result<Vec<Eval>>`                            | Same loop with an explicit concurrency level. `max_concurrency` LM calls run in flight at once (clamped to at least 1). Use `1` for strictly sequential evaluation on rate-limited providers                                                                        |
| `DEFAULT_EVAL_CONCURRENCY`           | `const usize = 16`                                                                                           | Concurrency used by `evaluate_trainset`                                                                                                                                                                                                                             |
| `average_score`                      | `fn(evals: &[Eval]) -> f64`                                                                                  | Arithmetic mean of scores; returns `0.0` for an empty slice                                                                                                                                                                                                         |

Each example runs inside a trace capture scope and the metric receives that rollout's `Trace`. Metric evaluation itself happens outside the scope, so LM-as-judge metrics do not pollute the execution trace.

## Feedback helpers

Helper functions in `evaluate::feedback_helpers` construct `Eval`s with structured textual feedback for common domains. All return `Eval`.

| Function                     | Signature                                                                                                 | Purpose                                                                                                                                                                                                                                 |
| ---------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `retrieval_feedback`         | `(retrieved: &[impl AsRef<str>], expected: &[impl AsRef<str>], context_docs: Option<&[impl AsRef<str>]>)` | Document retrieval. Score is F1; feedback lists correctly retrieved, missed, and incorrectly retrieved documents with precision, recall, and F1. Precision is `0.0` when `retrieved` is empty; recall is `1.0` when `expected` is empty |
| `code_pipeline_feedback`     | `(stages: &[(CodeStage, StageResult)], final_score: f64)`                                                 | Code generation pipelines. Feedback reports each stage in order and stops at the first failure; the score is the caller-supplied `final_score`                                                                                          |
| `multi_objective_feedback`   | `(objectives: &HashMap<String, (f64, String)>, weights: Option<&HashMap<String, f64>>)`                   | Multi-objective evaluation. Score is the weighted average (default weight `1.0` per objective); feedback lists each objective, sorted by name, plus the aggregate                                                                       |
| `string_similarity_feedback` | `(predicted: &str, expected: &str)`                                                                       | String comparison. `1.0` for an exact trimmed match, `0.95` for a case-insensitive match, otherwise word-level F1 with missing and extra words listed                                                                                   |
| `classification_feedback`    | `(predicted_class: &str, expected_class: &str, confidence: Option<f64>)`                                  | Classification. `1.0` on exact class match, `0.0` otherwise; feedback names the expected and predicted classes and the confidence when given                                                                                            |

Supporting enums for `code_pipeline_feedback`:

| Enum          | Variants                               |
| ------------- | -------------------------------------- |
| `CodeStage`   | `Parse`, `Compile`, `Execute`, `Test`  |
| `StageResult` | `Success`, `Failure { error: String }` |

## Metrics and optimizers

Optimizers call the evaluation loop internally; the metric you hand them determines what they can do with the results.

| Metric style        | Constructor                      | Sufficient for                                                                                             |
| ------------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Score-only          | `Eval::score(f64)`               | `COPRO`, `MIPROv2`, and any optimizer that ranks candidates by `average_score`                             |
| Score plus feedback | `Eval::with_feedback(f64, text)` | Required by `GEPA`, which feeds the textual feedback into its reflection step to guide evolutionary search |

The feedback helpers above exist mainly to serve GEPA: a metric that explains why a rollout scored low gives the reflection model something concrete to fix. See [Optimizers](/docs/components/optimizers) for the optimizer-side contract.

## See also

* [Optimizers](/docs/components/optimizers)
* [Traces](/docs/components/traces)
* [Data](/docs/components/data)
* [GEPA](/docs/optimizers/gepa)
* [Example: evaluate on HotpotQA](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/03-evaluate-hotpotqa.rs)
* [Example: optimize on HotpotQA](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/04-optimize-hotpotqa.rs)
* [Example: GEPA sentiment](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/09-gepa-sentiment.rs)
* [Example: GEPA LLM judge](https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/10-gepa-llm-judge.rs)