> ## Documentation Index
> Fetch the complete documentation index at: https://dsrs.herumbshandilya.com/llms.txt
> Use this file to discover all available pages before exploring further.

# GEPA with LLM-as-Judge

> Use an LLM judge to automatically generate rich feedback

This guide explains how to use an LLM judge to automatically generate rich textual feedback for GEPA optimization, making it easier to optimize complex tasks where manual feedback rules are hard to codify.

## The Pattern

Instead of manually writing feedback rules, use another LLM to evaluate the output and reasoning:

```
Task LM → generates answer + reasoning
    ↓
Judge LM → analyzes quality and provides feedback
    ↓
GEPA Reflection LM → reads feedback and improves prompt
    ↓
Better Task LM prompt
```

## Why Use LLM-as-a-Judge?

<Tabs>
  <Tab title="Good For">
    * Subjective quality assessment (writing style, helpfulness, clarity)
    * Complex reasoning evaluation (is the logic sound?)
    * Tasks where rules are hard to codify
    * Analyzing reasoning quality beyond just answer correctness
  </Tab>

  <Tab title="When to Avoid">
    * You have deterministic checks (unit tests, schema validation)
    * Verifiable correctness (code compilation, exact matches)
    * Need fast/cheap evaluation
    * Simple binary pass/fail
  </Tab>
</Tabs>

## Complete Example Walkthrough

<Card title="Full Working Example" icon="code" href="https://github.com/krypticmouse/DSRs/blob/main/crates/dspy-rs/examples/10-gepa-llm-judge.rs">
  See the complete implementation with step-by-step comments
</Card>

### 1. Task Signature with Reasoning

```rust theme={null}
#[derive(Signature, Clone, Debug)]
/// Solve math word problems step by step.
struct MathWordProblem {
    #[input]
    pub problem: String,

    #[output]
    pub reasoning: String,  // We want to optimize this too

    #[output]
    pub answer: String,
}
```

### 2. Judge Signature

```rust theme={null}
#[derive(Signature, Clone, Debug)]
/// You are an expert math teacher evaluating student work.
struct MathJudge {
    
    #[input]
    pub problem: String,
    
    #[input]
    pub expected_answer: String,
    
    #[input]
    pub student_answer: String,
    
    #[input]
    pub student_reasoning: String,
    
    #[output(desc = "Detailed evaluation of the work")]
    pub evaluation: String,  // This becomes the feedback
}
```

### 3. Optimized Module

```rust theme={null}
#[derive(Builder, facet::Facet)]
#[facet(crate = facet)]
struct MathSolver {
    #[builder(default = Predict::<MathWordProblem>::new())]
    solver: Predict<MathWordProblem>,  // This gets optimized
}
```

### 4. TypedMetric with Judge

```rust theme={null}
struct LlmJudgeMetric {
    judge: Predict<MathJudge>,
}

impl TypedMetric<MathWordProblem, MathSolver> for LlmJudgeMetric {
    async fn evaluate(
        &self,
        example: &Example<MathWordProblem>,
        prediction: &Predicted<<MathSolver as Module>::Output>,
    ) -> Result<MetricOutcome> {
        let problem = example.input.problem.clone();
        let expected = example.output.answer.clone();

        let student_answer = prediction.answer.clone();
        let student_reasoning = prediction.reasoning.clone();
        let exact_match = student_answer.trim() == expected.trim();

        let judge_output = self
            .judge
            .call(MathJudgeInput {
                problem: problem.clone(),
                expected_answer: expected.clone(),
                student_answer: student_answer.clone(),
                student_reasoning: student_reasoning.clone(),
            })
            .await;

        let (score, evaluation_text) = match judge_output {
            Ok(evaluation) => {
                let evaluation_text = evaluation.evaluation.clone();
                let evaluation_lc = evaluation_text.to_lowercase();
                let good_reasoning =
                    evaluation_lc.contains("sound reasoning")
                    || evaluation_lc.contains("correct approach")
                    || evaluation_lc.contains("clear");
                let partial_reasoning =
                    evaluation_lc.contains("partially")
                    || evaluation_lc.contains("good start")
                    || evaluation_lc.contains("minor arithmetic")
                    || evaluation_lc.contains("close");

                let score = match (exact_match, good_reasoning, partial_reasoning) {
                    (true, true, _) => 1.0,
                    (true, false, _) => 0.7,
                    (false, true, _) | (false, _, true) => 0.3,
                    (false, false, false) => 0.0,
                };
                (score, evaluation_text)
            }
            Err(err) => {
                let fallback = format!(
                    "judge call failed: {err}; expected={expected}; predicted={student_answer}"
                );
                ((exact_match as u8 as f32), fallback)
            }
        };

        let feedback = FeedbackMetric::new(
            score,
            format!(
                "problem={problem}\nexpected={expected}\npredicted={student_answer}\njudge={evaluation_text}"
            ),
        );

        Ok(MetricOutcome::with_feedback(score, feedback))
    }
}
```

`GEPA` itself does not own a special `feedback_metric` hook anymore.
The feedback function lives in your `TypedMetric` implementation, and GEPA enforces that every evaluation returns `MetricOutcome::with_feedback(...)`.
That keeps the optimizer generic while preserving full judge-driven behavior.

## Key Benefits

<AccordionGroup>
  <Accordion title="Catches Lucky Guesses">
    ```
    Answer: Correct (1.0)
    But reasoning: "I just multiplied random numbers"
    Score: 0.7 (penalized for bad reasoning)
    ```

    The judge identifies when the model got the right answer for the wrong reasons.
  </Accordion>

  <Accordion title="Rewards Partial Progress">
    ```
    Answer: Wrong
    But reasoning: "Correct approach, arithmetic error in final step"
    Score: 0.3 (partial credit)
    ```

    The judge recognizes valid methodology even when the final answer is wrong.
  </Accordion>

  <Accordion title="Identifies Systematic Issues">
    The judge notices patterns like:

    * "Model consistently skips showing intermediate steps"
    * "Model confuses similar concepts (area vs perimeter)"
    * "Model doesn't check units in answers"

    GEPA's reflection can then say:

    > "Add explicit instruction to show all intermediate steps and verify units"
  </Accordion>
</AccordionGroup>

## Cost Considerations

<Warning>
  LLM judges double your evaluation cost since every prediction requires both a task LM call and a judge LM call.
</Warning>

Budget accordingly:

```rust theme={null}
GEPA::builder()
    .num_iterations(3)           // Fewer iterations
    .minibatch_size(3)           // Smaller batches
    .max_lm_calls(Some(100))  // Explicit limit
    .build()
```

### Optimization Tips

* Use a cheaper model for judging (gpt-4o-mini vs gpt-4)
* Judge only failed examples (not ones that passed)
* Cache judge evaluations for identical outputs
* Use parallel evaluation to reduce wall-clock time

## Hybrid Approach

Best results often come from combining explicit checks with LLM judging:

```rust theme={null}
impl TypedMetric<MySignature, MyModule> for HybridMetric {
    async fn evaluate(
        &self,
        example: &Example<MySignature>,
        prediction: &Predicted<<MyModule as Module>::Output>,
    ) -> Result<MetricOutcome> {
        let mut score = 1.0;
        let mut feedback_parts = vec![];

        // Explicit checks first (fast, cheap, deterministic)
        if !is_valid_json(&prediction.result_json) {
            feedback_parts.push("Invalid JSON format".to_string());
            score = 0.0;
        }

        if score > 0.0 && missing_required_fields(&prediction.result_json) {
            feedback_parts.push("Missing fields: user_id, timestamp".to_string());
            score *= 0.5;
        }

        // Optional judge pass for qualitative scoring
        if score > 0.0 {
            let judge_feedback = self.judge_quality(example, prediction).await?;
            if judge_feedback.to_lowercase().contains("low quality") {
                score *= 0.7;
            }
            feedback_parts.push(judge_feedback);
        }

        let feedback = FeedbackMetric::new(score, feedback_parts.join("\n"));
        Ok(MetricOutcome::with_feedback(score, feedback))
    }
}
```

## Example Evolution

When you run the example, GEPA will evolve prompts based on judge feedback:

<Steps>
  <Step title="Baseline">
    Instruction: "Solve the math word problem step by step"

    Result: Some solutions skip steps

    Judge: "Reasoning incomplete, jumped from step 2 to answer"
  </Step>

  <Step title="After GEPA">
    Instruction: "Solve step by step. Show ALL intermediate calculations. Label each step clearly."

    Result: Complete solutions with all steps shown

    Judge: "Sound reasoning, all steps shown clearly"
  </Step>
</Steps>

The judge's analysis becomes the signal that drives prompt improvement.

## Running the Example

```bash theme={null}
OPENAI_API_KEY=your_key cargo run --example 10-gepa-llm-judge
```

This will show:

1. Baseline performance
2. Judge evaluations during optimization
3. How feedback evolves the prompt
4. Final test with judge analysis