GEPA with LLM-as-Judge

This guide explains how to use an LLM judge to automatically generate rich textual feedback for GEPA optimization, making it easier to optimize complex tasks where manual feedback rules are hard to codify.

The Pattern

Instead of manually writing feedback rules, use another LLM to evaluate the output and reasoning:

Task LM → generates answer + reasoning
    ↓
Judge LM → analyzes quality and provides feedback
    ↓
GEPA Reflection LM → reads feedback and improves prompt
    ↓
Better Task LM prompt

Why Use LLM-as-a-Judge?

Good For
When to Avoid

Subjective quality assessment (writing style, helpfulness, clarity)
Complex reasoning evaluation (is the logic sound?)
Tasks where rules are hard to codify
Analyzing reasoning quality beyond just answer correctness

Complete Example Walkthrough

Full Working Example

See the complete implementation with step-by-step comments

1. Task Signature with Reasoning

#[derive(Signature, Clone, Debug)]
/// Solve math word problems step by step.
struct MathWordProblem {
    #[input]
    pub problem: String,

    #[output]
    pub reasoning: String,  // We want to optimize this too

    #[output]
    pub answer: String,
}

2. Judge Signature

#[derive(Signature, Clone, Debug)]
/// You are an expert math teacher evaluating student work.
struct MathJudge {
    
    #[input]
    pub problem: String,
    
    #[input]
    pub expected_answer: String,
    
    #[input]
    pub student_answer: String,
    
    #[input]
    pub student_reasoning: String,
    
    #[output(desc = "Detailed evaluation of the work")]
    pub evaluation: String,  // This becomes the feedback
}

3. Optimized Module

#[derive(Builder, facet::Facet)]
#[facet(crate = facet)]
struct MathSolver {
    #[builder(default = Predict::<MathWordProblem>::new())]
    solver: Predict<MathWordProblem>,  // This gets optimized
}

4. TypedMetric with Judge

struct LlmJudgeMetric {
    judge: Predict<MathJudge>,
}

impl TypedMetric<MathWordProblem, MathSolver> for LlmJudgeMetric {
    async fn evaluate(
        &self,
        example: &Example<MathWordProblem>,
        prediction: &Predicted<<MathSolver as Module>::Output>,
    ) -> Result<MetricOutcome> {
        let problem = example.input.problem.clone();
        let expected = example.output.answer.clone();

        let student_answer = prediction.answer.clone();
        let student_reasoning = prediction.reasoning.clone();
        let exact_match = student_answer.trim() == expected.trim();

        let judge_output = self
            .judge
            .call(MathJudgeInput {
                problem: problem.clone(),
                expected_answer: expected.clone(),
                student_answer: student_answer.clone(),
                student_reasoning: student_reasoning.clone(),
            })
            .await;

        let (score, evaluation_text) = match judge_output {
            Ok(evaluation) => {
                let evaluation_text = evaluation.evaluation.clone();
                let evaluation_lc = evaluation_text.to_lowercase();
                let good_reasoning =
                    evaluation_lc.contains("sound reasoning")
                    || evaluation_lc.contains("correct approach")
                    || evaluation_lc.contains("clear");
                let partial_reasoning =
                    evaluation_lc.contains("partially")
                    || evaluation_lc.contains("good start")
                    || evaluation_lc.contains("minor arithmetic")
                    || evaluation_lc.contains("close");

                let score = match (exact_match, good_reasoning, partial_reasoning) {
                    (true, true, _) => 1.0,
                    (true, false, _) => 0.7,
                    (false, true, _) | (false, _, true) => 0.3,
                    (false, false, false) => 0.0,
                };
                (score, evaluation_text)
            }
            Err(err) => {
                let fallback = format!(
                    "judge call failed: {err}; expected={expected}; predicted={student_answer}"
                );
                ((exact_match as u8 as f32), fallback)
            }
        };

        let feedback = FeedbackMetric::new(
            score,
            format!(
                "problem={problem}\nexpected={expected}\npredicted={student_answer}\njudge={evaluation_text}"
            ),
        );

        Ok(MetricOutcome::with_feedback(score, feedback))
    }
}

GEPA itself does not own a special feedback_metric hook anymore. The feedback function lives in your TypedMetric implementation, and GEPA enforces that every evaluation returns MetricOutcome::with_feedback(...). That keeps the optimizer generic while preserving full judge-driven behavior.

Key Benefits

Catches Lucky Guesses

Answer: Correct (1.0)
But reasoning: "I just multiplied random numbers"
Score: 0.7 (penalized for bad reasoning)

The judge identifies when the model got the right answer for the wrong reasons.

Rewards Partial Progress

Answer: Wrong
But reasoning: "Correct approach, arithmetic error in final step"
Score: 0.3 (partial credit)

The judge recognizes valid methodology even when the final answer is wrong.

Identifies Systematic Issues

The judge notices patterns like:

“Model consistently skips showing intermediate steps”
“Model confuses similar concepts (area vs perimeter)”
“Model doesn’t check units in answers”

GEPA’s reflection can then say:

“Add explicit instruction to show all intermediate steps and verify units”

Cost Considerations

LLM judges double your evaluation cost since every prediction requires both a task LM call and a judge LM call.

Budget accordingly:

GEPA::builder()
    .num_iterations(3)           // Fewer iterations
    .minibatch_size(3)           // Smaller batches
    .max_lm_calls(Some(100))  // Explicit limit
    .build()

Optimization Tips

Use a cheaper model for judging (gpt-4o-mini vs gpt-4)
Judge only failed examples (not ones that passed)
Cache judge evaluations for identical outputs
Use parallel evaluation to reduce wall-clock time

Hybrid Approach

Best results often come from combining explicit checks with LLM judging:

impl TypedMetric<MySignature, MyModule> for HybridMetric {
    async fn evaluate(
        &self,
        example: &Example<MySignature>,
        prediction: &Predicted<<MyModule as Module>::Output>,
    ) -> Result<MetricOutcome> {
        let mut score = 1.0;
        let mut feedback_parts = vec![];

        // Explicit checks first (fast, cheap, deterministic)
        if !is_valid_json(&prediction.result_json) {
            feedback_parts.push("Invalid JSON format".to_string());
            score = 0.0;
        }

        if score > 0.0 && missing_required_fields(&prediction.result_json) {
            feedback_parts.push("Missing fields: user_id, timestamp".to_string());
            score *= 0.5;
        }

        // Optional judge pass for qualitative scoring
        if score > 0.0 {
            let judge_feedback = self.judge_quality(example, prediction).await?;
            if judge_feedback.to_lowercase().contains("low quality") {
                score *= 0.7;
            }
            feedback_parts.push(judge_feedback);
        }

        let feedback = FeedbackMetric::new(score, feedback_parts.join("\n"));
        Ok(MetricOutcome::with_feedback(score, feedback))
    }
}

Example Evolution

When you run the example, GEPA will evolve prompts based on judge feedback:

Baseline

Instruction: “Solve the math word problem step by step”Result: Some solutions skip stepsJudge: “Reasoning incomplete, jumped from step 2 to answer”

After GEPA

Instruction: “Solve step by step. Show ALL intermediate calculations. Label each step clearly.”Result: Complete solutions with all steps shownJudge: “Sound reasoning, all steps shown clearly”

The judge’s analysis becomes the signal that drives prompt improvement.

Running the Example

OPENAI_API_KEY=your_key cargo run --example 10-gepa-llm-judge

This will show:

Baseline performance
Judge evaluations during optimization
How feedback evolves the prompt
Final test with judge analysis

Get Started

Building Blocks

Data

Optimizers

GEPA with LLM-as-Judge

The Pattern

Why Use LLM-as-a-Judge?

Complete Example Walkthrough

Full Working Example

1. Task Signature with Reasoning

2. Judge Signature

3. Optimized Module

4. TypedMetric with Judge

Key Benefits

Cost Considerations

Optimization Tips

Hybrid Approach

Example Evolution

Running the Example

​The Pattern

​Why Use LLM-as-a-Judge?

​Complete Example Walkthrough

Full Working Example

​1. Task Signature with Reasoning

​2. Judge Signature

​3. Optimized Module

​4. TypedMetric with Judge

​Key Benefits

​Cost Considerations

​Optimization Tips

​Hybrid Approach

​Example Evolution

​Running the Example

The Pattern

Why Use LLM-as-a-Judge?

Complete Example Walkthrough

1. Task Signature with Reasoning

2. Judge Signature

3. Optimized Module

4. TypedMetric with Judge

Key Benefits

Cost Considerations

Optimization Tips

Hybrid Approach

Example Evolution

Running the Example