Evaluation

Evaluation is how you tell an optimizer what “good” means. The evaluate module provides a metric trait, a result type, a concurrent evaluation loop, and helper functions that build rich textual feedback. A metric sees fully typed data on both sides: the trainset row with all its gold data and the prediction as the module actually produced it.

Writing and running a metric

A minimal exact-match metric for a QA task, over a row struct trainset:

use anyhow::Result;
use dspy_rs::{Eval, Example, Predict, Predicted, Trace, TypedMetric};

#[derive(Example, Clone, Debug, serde::Serialize)]
struct QARow {
    question: String,
    answer: String,
}

struct ExactMatchMetric;

impl TypedMetric<QARow, Predict<QA>> for ExactMatchMetric {
    async fn evaluate(
        &self,
        example: &QARow,
        prediction: &Predicted<QAOutput>,
        _trace: Option<&Trace>,
    ) -> Result<Eval> {
        let expected = example.answer.trim().to_lowercase();
        let actual = prediction.answer.trim().to_lowercase();
        Ok(Eval::score((expected == actual) as u8 as f64))
    }
}

evaluate_trainset runs the module on every example, scores each rollout with the metric, and average_score reduces the results to one number:

use dspy_rs::{average_score, evaluate_trainset};

let evals = evaluate_trainset(&module, &trainset, &ExactMatchMetric).await?;
println!("Average: {:.3}", average_score(&evals));

TypedMetric

TypedMetric<E, M> is the metric trait. It is generic over E (the trainset row — any struct you like, see Data) and M: Module, so the metric sees fully typed data: the row with all its fields, and the prediction as the module actually produced it (for example WithReasoning<QAOutput> from ChainOfThought). Because the metric receives the row rather than a signature-shaped pair, gold data need not fit the module’s output type — a row can carry metric-only fields the module never sees, such as HotpotQA supporting facts.

pub trait TypedMetric<E, M>: Send + Sync
where
    M: Module,
{
    async fn evaluate(
        &self,
        example: &E,
        prediction: &Predicted<M::Output>,
        trace: Option<&Trace>,
    ) -> Result<Eval>;
}

Argument	Meaning
`example`	The trainset row under evaluation: the full `E`, including gold labels and metric-only fields (the ground truth)
`prediction`	The module’s `Predicted<M::Output>`
`trace`	The rollout’s execution `Trace` when the caller captured one. The evaluation loop always passes `Some`; direct callers may pass `None`. Slice it per component with `trace.for_component("retriever")`

Return Eval::score(f64) for a numerical score, Eval::with_feedback(f64, text) to also explain why. Scores are 0.0 to 1.0 by convention.

Eval and Rollout

Eval is defined in the trace module and re-exported by evaluate.

Type	Definition	Notes
`Eval`	`{ score: f64, feedback: Option<String> }`	Constructors: `Eval::score(score)` and `Eval::with_feedback(score, feedback)`
`Rollout`	`(Eval, Trace)`	One evaluated rollout: the metric result plus the execution trace that produced it, with `Trace::outcome` filled in (serialized output, the `Eval`, duration)

The public evaluation entry points return Vec<Eval>; the traced Rollout path is what optimizers consume internally.

Evaluation functions

Item	Signature	Behavior
`evaluate_trainset`	`async fn(module: &M, trainset: &[E], metric: &MT) -> Result<Vec<Eval>>` where `E: ToInput<M::Input> + Sync`	Runs the module on every row (projected via `to_input()`) and scores each with the metric. Results come back in trainset order. Any `Module::call` or `TypedMetric::evaluate` failure propagates immediately; for fault-tolerant batching use `forward_all` instead
`evaluate_trainset_with_concurrency`	`async fn(module, trainset, metric, max_concurrency: usize) -> Result<Vec<Eval>>`	Same loop with an explicit concurrency level. `max_concurrency` LM calls run in flight at once (clamped to at least 1). Use `1` for strictly sequential evaluation on rate-limited providers
`DEFAULT_EVAL_CONCURRENCY`	`const usize = 16`	Concurrency used by `evaluate_trainset`
`average_score`	`fn(evals: &[Eval]) -> f64`	Arithmetic mean of scores; returns `0.0` for an empty slice

Each example runs inside a trace capture scope and the metric receives that rollout’s Trace. Metric evaluation itself happens outside the scope, so LM-as-judge metrics do not pollute the execution trace.

Feedback helpers

Helper functions in evaluate::feedback_helpers construct Evals with structured textual feedback for common domains. All return Eval.

Function	Signature	Purpose
`retrieval_feedback`	`(retrieved: &[impl AsRef<str>], expected: &[impl AsRef<str>], context_docs: Option<&[impl AsRef<str>]>)`	Document retrieval. Score is F1; feedback lists correctly retrieved, missed, and incorrectly retrieved documents with precision, recall, and F1. Precision is `0.0` when `retrieved` is empty; recall is `1.0` when `expected` is empty
`code_pipeline_feedback`	`(stages: &[(CodeStage, StageResult)], final_score: f64)`	Code generation pipelines. Feedback reports each stage in order and stops at the first failure; the score is the caller-supplied `final_score`
`multi_objective_feedback`	`(objectives: &HashMap<String, (f64, String)>, weights: Option<&HashMap<String, f64>>)`	Multi-objective evaluation. Score is the weighted average (default weight `1.0` per objective); feedback lists each objective, sorted by name, plus the aggregate
`string_similarity_feedback`	`(predicted: &str, expected: &str)`	String comparison. `1.0` for an exact trimmed match, `0.95` for a case-insensitive match, otherwise word-level F1 with missing and extra words listed
`classification_feedback`	`(predicted_class: &str, expected_class: &str, confidence: Option<f64>)`	Classification. `1.0` on exact class match, `0.0` otherwise; feedback names the expected and predicted classes and the confidence when given

Supporting enums for code_pipeline_feedback:

Enum	Variants
`CodeStage`	`Parse`, `Compile`, `Execute`, `Test`
`StageResult`	`Success`, `Failure { error: String }`

Metrics and optimizers

Optimizers call the evaluation loop internally; the metric you hand them determines what they can do with the results.

Metric style	Constructor	Sufficient for
Score-only	`Eval::score(f64)`	`COPRO`, `MIPROv2`, and any optimizer that ranks candidates by `average_score`
Score plus feedback	`Eval::with_feedback(f64, text)`	Required by `GEPA`, which feeds the textual feedback into its reflection step to guide evolutionary search

The feedback helpers above exist mainly to serve GEPA: a metric that explains why a rollout scored low gives the reflection model something concrete to fix. See Optimizers for the optimizer-side contract.

Get Started

Building Blocks

Programs as Data

Tools & Agents

Evaluation & Optimization

Utilities

Writing and running a metric

TypedMetric

Eval and Rollout

Evaluation functions

Feedback helpers

Metrics and optimizers

See also

​Writing and running a metric

​TypedMetric

​Eval and Rollout

​Evaluation functions

​Feedback helpers

​Metrics and optimizers

​See also

Writing and running a metric

TypedMetric

Eval and Rollout

Evaluation functions

Feedback helpers

Metrics and optimizers

See also