ToolsPromptsSkillsWorkflowsBlogAbout

LLM-as-Judge — automate evaluation with AI scoring

Use a strong LLM to score model outputs against rubrics, compare A/B variants, and generate qualitative feedback at scale without human annotators.

Evaluation4.7
LLM-as-Judge — automate evaluation with AI scoring preview

Prompt template

You are an expert evaluator. Score the following model output on a scale of 1-5 for each criterion. Provide a brief justification for each score.

Criteria: {rubric}

Input: {input}

Output to evaluate: {output}

Reference answer (if available): {reference}

Scores:

How to use

  1. 1Define a clear rubric with 3-5 criteria and a 1-5 or 1-10 scoring scale
  2. 2Prepare an evaluation dataset of 20-100 representative inputs with optional reference answers
  3. 3Run the target model on all evaluation inputs to collect outputs
  4. 4Send each input-output pair (and reference if available) to a judge LLM with the rubric
  5. 5Aggregate scores across the dataset, compute mean and confidence intervals per criterion
  6. 6For pairwise A/B comparison, ask the judge to pick the better output with justification (reduces scale bias)

Pro tips

  • Pairwise comparison (which is better?) is more reliable than absolute scoring (score 1-5) due to position bias
  • Use a stronger model as judge than the model being evaluated (e.g. GPT-4 judging GPT-3.5 outputs)
  • Calibrate the judge against human annotations on a subset — aim for 0.7+ Cohen's kappa
  • Always include a 'no preference' option in pairwise judging to detect ties

Works best with

promptfoolangsmithragasdeepeval

Ratings & Reviews

Loading reviews…

Share your experience

Select a rating

Related Skills

Prompt Versioning and A/B Testing — iterate with data
Optimization

Prompt Versioning and A/B Testing — iterate with data

Track prompt changes, run controlled A/B tests against evaluation datasets, and deploy the best-performing version with confidence.

A/B TestingVersioningEvaluation
4.4
Hallucination Detection — identify and reduce fabricated content
Evaluation

Hallucination Detection — identify and reduce fabricated content

Deploy a multi-layered strategy: factuality scoring, citation verification, consistency checks, and self-contradiction detection to catch LLM fabrications.

HallucinationFactualityDetection
4.4