Skip to content

Automated Evaluation and Regression Testing

An evaluation you run by hand is an evaluation you skip under deadline pressure. This page shows how to make evaluation part of the delivery pipeline: every change to a prompt, model, retrieval setting or tool runs against a golden dataset, the scores are compared with an approved baseline, and the pull request is blocked when quality drops. The running example is the support assistant of an invented OrdersPortal app, which answers questions such as “Where is order ORD-1042?” from order data and help articles.

The samples are Python 3.12 or later. The scripts that only process JSON (regression_check.py, check_thresholds.py, baseline.py) were run with Python 3.13 and their output is shown. The samples that call a model need API keys and were not run here; they are checked against DeepEval 4.3 and the LangSmith SDK 0.15 signatures.

Topic What it answers
Why automate evaluation What a pipeline catches that manual checks miss
Pipeline architecture The stages from push to merge decision
GitHub Actions workflow A pull-request job that runs evals and posts scores
Evaluation runner Turning a golden dataset into metric scores with DeepEval
Threshold gate Failing the build when a metric is below its floor
Regression testing Why AI systems regress and the baseline lifecycle
Baselines Storing approved scores with the context that produced them
Regression detection Comparing a candidate with the baseline per metric
Pairwise evaluation Choosing between two models or prompts with a judge
Prompt regression tests Expressing quality rules as pytest tests
LangSmith experiments Hosted datasets, experiments and history
Scheduled evaluation Catching drift when nothing in your repo changed
Keeping evals fast and cheap Tiered datasets, caching and concurrency
Accepting a regression When a drop is the right trade-off

Automated pipelines enforce quality gates the same way unit tests do: a change either passes or it does not. For language-model features that gives you three things manual review cannot:

  • A prompt edit that fixes one intent but breaks another is caught before release, not from customer tickets.
  • A model upgrade decision is based on your data (“Terra answers 4 points more faithfully on our 200 support questions”), not on a vendor leaderboard.
  • Quality trends are visible over time without anyone keeping a spreadsheet.

The difference from ordinary tests is that outputs are probabilistic and graded, so the gate compares scores with thresholds and baselines instead of asserting exact strings.

Developer opens a pull request that touches prompts, retrieval or agent code
│
▼
CI job: load golden dataset ──► run each case through the app ──► score with metrics
│
├── absolute gate: every metric above its floor?
├── regression gate: no metric dropped more than its tolerance vs the baseline?
│
├── both pass ──► merge allowed
└── either fails ──► merge blocked, report posted on the PR
│
▼
Scores recorded (LangSmith experiment or your own history file)

Two gates matter. The absolute gate says “faithfulness must never be below 0.75”. The regression gate says “faithfulness must not drop by more than 0.05 from the approved baseline”, which catches a slide from 0.92 to 0.80 that the absolute gate would wave through.

The job runs only when files that affect model behavior change, so documentation-only pull requests stay fast. Secrets come from repository or environment secrets, never from the workflow file.

.github/workflows/ai-eval.yml
name: AI evaluation
on:
pull_request:
paths:
- "src/prompts/**"
- "src/rag/**"
- "src/agents/**"
- "evals/**"
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # needed to post the score comment
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: "3.13"
cache: "pip"
- name: Install dependencies
run: pip install -r evals/requirements.txt # pins deepeval, langsmith, openai
- name: Run evaluation suite
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
LANGSMITH_TRACING: "true"
LANGSMITH_PROJECT: "ordersportal-ci-evals"
run: |
python evals/run_evals.py \
--dataset evals/golden_dataset.jsonl \
--output eval_results.json \
--threshold 0.75
- name: Absolute threshold gate
run: python evals/check_thresholds.py eval_results.json
- name: Regression gate
run: python evals/regression_check.py eval_results.json evals/baseline.json
- name: Post results to the pull request
if: always()
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (!fs.existsSync('eval_results.json')) return;
const r = JSON.parse(fs.readFileSync('eval_results.json'));
const rows = Object.entries(r.scores).map(([m, s]) =>
`| ${m} | ${s.toFixed(3)} | ${s >= r.thresholds[m] ? 'pass' : 'FAIL'} |`);
await github.rest.issues.createComment({
...context.repo,
issue_number: context.issue.number,
body: ['## AI evaluation results', '', '| Metric | Score | Status |', '|---|---|---|', ...rows].join('\n'),
});

The runner loads the golden dataset (one JSON object per line), calls the application for each case, and scores the outputs. DeepEval’s evaluate() returns an EvaluationResult whose test_results each carry metrics_data with a name, score and success flag, so the per-metric score is the mean over test cases. A common bug is to compute one pass rate and copy it to every metric, which hides which dimension regressed.

# evals/run_evals.py — fragment: needs your application and an OpenAI key for the judge
import argparse
import json
from collections import defaultdict
from pathlib import Path
from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from ordersportal.support import answer_question # your application code
def load_dataset(path: str) -> list[dict]:
return [json.loads(line) for line in Path(path).read_text(encoding="utf-8").splitlines() if line]
def score(dataset: list[dict]) -> dict[str, float]:
cases = []
for item in dataset:
result = answer_question(item["input"])
cases.append(LLMTestCase(
input=item["input"],
actual_output=result["answer"],
expected_output=item["expected_output"],
retrieval_context=result.get("context", []),
))
metrics = [AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.75)]
outcome = evaluate(test_cases=cases, metrics=metrics)
per_metric: dict[str, list[float]] = defaultdict(list)
for test_result in outcome.test_results:
for data in test_result.metrics_data or []:
per_metric[data.name].append(data.score or 0.0)
return {name: sum(vals) / len(vals) for name, vals in per_metric.items()}
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--dataset", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--threshold", type=float, default=0.75)
args = parser.parse_args()
scores = score(load_dataset(args.dataset))
output = {"scores": scores, "thresholds": {m: args.threshold for m in scores}}
Path(args.output).write_text(json.dumps(output, indent=2), encoding="utf-8")
print("Evaluation complete:", scores)

The absolute gate is deliberately boring: read the results file, compare each score with its floor, and exit non-zero on failure so the CI step fails.

# evals/check_thresholds.py — absolute quality gate: exit 1 if any metric is below its floor
import json
import sys
from pathlib import Path
def check(results_file: str) -> bool:
results = json.loads(Path(results_file).read_text(encoding="utf-8"))
failures = [
f"{metric}: {score:.3f} < {results['thresholds'][metric]:.3f}"
for metric, score in results["scores"].items()
if score < results["thresholds"][metric]
]
for line in failures:
print("FAIL", line)
if not failures:
print("All metrics passed their thresholds")
return not failures
if __name__ == "__main__":
sys.exit(0 if check(sys.argv[1]) else 1)

With a results file where faithfulness scored 0.731 against a floor of 0.75:

$ python check_thresholds.py eval_results.json; echo "exit=$?"
FAIL FaithfulnessMetric: 0.731 < 0.750
exit=1

Pitfalls:

  • One global threshold. Faithfulness and tone do not deserve the same floor. Store a threshold per metric and review them like code.
  • Judge failures counted as passes. If the judge call errors, DeepEval can record a missing score. Treat a missing score as zero (as the runner above does) or fail the job, never skip the case silently.

In ordinary software a regression is a change that breaks something that used to work. AI systems regress for reasons that never appear in your diff:

  • the provider updates a model behind an alias, or retires the snapshot you pinned;
  • a prompt edit improves one task and quietly degrades another;
  • a retrieval change (chunk size, embedding model, top-k) shifts which documents reach the model;
  • a new tool or system message changes how the model plans.

Regression testing is the discipline of detecting these changes before users do. The lifecycle has four steps: run the evaluation on the current system and store the scores as an approved baseline; make the change; run the same evaluation on the same dataset; compare each metric with the baseline and decide: proceed, investigate, revert, or deliberately accept and record a new baseline. The comparison is only meaningful when the dataset, judge model and metric definitions are the same on both sides, which is why the baseline stores them.

A baseline is a set of scores plus the context that produced them: model ID, prompt version, dataset hash and who approved it. Hashing the dataset stops a common mistake, comparing scores from two different datasets after someone added cases.

# baseline.py — store and load approved baselines, keyed by model, prompt version and dataset
import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
BASELINE_FILE = Path("eval_baselines.json")
def dataset_hash(path: str) -> str:
return hashlib.sha256(Path(path).read_bytes()).hexdigest()[:12]
def save_baseline(scores: dict[str, float], model: str, prompt_version: str,
dataset_path: str, approved_by: str, notes: str = "") -> dict:
baselines = json.loads(BASELINE_FILE.read_text()) if BASELINE_FILE.exists() else []
entry = {
"timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"),
"model": model,
"prompt_version": prompt_version,
"dataset_hash": dataset_hash(dataset_path),
"scores": scores,
"approved_by": approved_by,
"notes": notes,
}
baselines.append(entry)
BASELINE_FILE.write_text(json.dumps(baselines, indent=2))
return entry
def latest_baseline(model: str, prompt_version: str, dataset_path: str) -> dict | None:
if not BASELINE_FILE.exists():
return None
wanted = dataset_hash(dataset_path)
matches = [b for b in json.loads(BASELINE_FILE.read_text())
if b["model"] == model and b["prompt_version"] == prompt_version
and b["dataset_hash"] == wanted]
return matches[-1] if matches else None
if __name__ == "__main__":
Path("golden_dataset.jsonl").write_text(
'{"input": "Where is order ORD-1042?", "expected_output": "Shipped on 12 Nov"}\n')
save_baseline({"faithfulness": 0.91}, "gpt-6.1-terra", "support-v7",
"golden_dataset.jsonl", approved_by="qa-lead")
found = latest_baseline("gpt-6.1-terra", "support-v7", "golden_dataset.jsonl")
print(found["prompt_version"], found["scores"], found["dataset_hash"] == dataset_hash("golden_dataset.jsonl"))
support-v7 {'faithfulness': 0.91} True

Keep the baseline file in version control next to the dataset, so a baseline change shows up in review like any other change. datetime.utcnow(), which older examples use, is deprecated since Python 3.12; use datetime.now(timezone.utc).

The regression gate compares each metric with the baseline and applies a per-metric tolerance. A tolerance is necessary because LLM-judged scores move a little between identical runs; set it from the spread you observe when you run the unchanged system several times.

# regression_check.py — compare a candidate run with the stored baseline
REGRESSION_THRESHOLDS = {
"answer_relevancy": 0.03, # fail if the score drops by more than 0.03
"faithfulness": 0.05,
"accuracy": 0.02,
}
def detect_regression(baseline: dict[str, float], current: dict[str, float],
thresholds: dict[str, float] = REGRESSION_THRESHOLDS) -> dict:
regressions, improvements, unchanged = [], [], []
for metric, score in current.items():
if metric not in baseline:
continue
delta = score - baseline[metric]
tolerance = thresholds.get(metric, 0.03)
row = {"metric": metric, "baseline": baseline[metric], "current": score, "delta": delta}
if delta < -tolerance:
regressions.append(row)
elif delta > tolerance:
improvements.append(row)
else:
unchanged.append(metric)
return {"passed": not regressions, "regressions": regressions,
"improvements": improvements, "unchanged": unchanged}
def format_report(result: dict) -> str:
lines = ["Regression report: " + ("PASS" if result["passed"] else "FAIL")]
for r in result["regressions"]:
lines.append(f" - {r['metric']}: {r['baseline']:.3f} -> {r['current']:.3f} ({r['delta']:+.3f})")
for r in result["improvements"]:
lines.append(f" + {r['metric']}: {r['baseline']:.3f} -> {r['current']:.3f} ({r['delta']:+.3f})")
if result["unchanged"]:
lines.append(" = within tolerance: " + ", ".join(result["unchanged"]))
return "\n".join(lines)
if __name__ == "__main__":
baseline = {"answer_relevancy": 0.86, "faithfulness": 0.91, "accuracy": 0.82}
candidate = {"answer_relevancy": 0.88, "faithfulness": 0.84, "accuracy": 0.86}
print(format_report(detect_regression(baseline, candidate)))
Regression report: FAIL
- faithfulness: 0.910 -> 0.840 (-0.070)
+ accuracy: 0.820 -> 0.860 (+0.040)
= within tolerance: answer_relevancy

In CI the __main__ block loads the two dictionaries from eval_results.json and baseline.json (the paths the workflow passes) instead of the demo values. The candidate is more accurate but less faithful to its sources, which for a support assistant means more confident answers that the order data does not support. A single averaged score would have hidden that.

Pitfalls:

  • Tolerance smaller than noise. If identical runs vary by 0.04, a 0.02 tolerance produces flaky failures and teams learn to re-run until green. Measure the noise first.
  • New metrics with no baseline. The function skips them; report them separately so they are not mistaken for passes.

When you choose between two models or two prompt versions, absolute scores are often too coarse. Pairwise evaluation shows a judge model both answers and asks which is better. Run each case more than once and take the majority, swap the A and B positions on alternate runs (judges favor the first answer), and keep the judge model fixed for the whole comparison.

# fragment: needs an OpenAI key and your own call_model() function
from collections import Counter
from openai import OpenAI
client = OpenAI()
JUDGE = "gpt-6.1-terra"
def judge(question: str, expected: str, first: str, second: str) -> str:
prompt = (f"Question: {question}\nReference answer: {expected}\n\n"
f"Response 1:\n{first}\n\nResponse 2:\n{second}\n\n"
"Which response is more accurate and complete? Reply with only 1, 2 or TIE.")
reply = client.chat.completions.create(
model=JUDGE, messages=[{"role": "user", "content": prompt}], max_tokens=5, temperature=0,
).choices[0].message.content.strip().upper()
return reply if reply in ("1", "2", "TIE") else "TIE"
def compare(cases: list[dict], model_a: str, model_b: str, runs: int = 3) -> Counter:
tally = Counter()
for case in cases:
votes = []
for i in range(runs):
a, b = call_model(model_a, case["input"]), call_model(model_b, case["input"])
if i % 2 == 0: # alternate positions to cancel position bias
v = judge(case["input"], case["expected"], a, b)
votes.append({"1": "A", "2": "B"}.get(v, "TIE"))
else:
v = judge(case["input"], case["expected"], b, a)
votes.append({"1": "B", "2": "A"}.get(v, "TIE"))
tally[Counter(votes).most_common(1)[0][0]] += 1
return tally # e.g. Counter({'A': 112, 'B': 61, 'TIE': 27})

Quality rules can live in the normal test suite, which makes them visible to developers who never open an eval dashboard. Mark them so they run in the eval job, not on every local pytest run, because they cost money and time.

# tests/test_prompt_regression.py — fragment: run_prompt_against_dataset is your own helper
import json
import pytest
from ordersportal.prompts import SUPPORT_PROMPT_V7, SUPPORT_PROMPT_V8
from ordersportal.evals import run_prompt_against_dataset
BASELINE = {"answer_relevancy": 0.86, "faithfulness": 0.91}
TOLERANCE = 0.03
@pytest.fixture(scope="session")
def golden_dataset():
with open("evals/golden_dataset.jsonl", encoding="utf-8") as f:
return [json.loads(line) for line in f if line.strip()]
@pytest.mark.eval
def test_v8_does_not_regress(golden_dataset):
scores = run_prompt_against_dataset(SUPPORT_PROMPT_V8, golden_dataset)
for metric, baseline in BASELINE.items():
assert scores[metric] >= baseline - TOLERANCE, f"{metric} regressed to {scores[metric]:.3f}"

Run with pytest -m eval in CI and register the eval marker in pyproject.toml so pytest does not warn about it.

LangSmith stores datasets and runs each evaluation as an experiment you can compare in its UI, which gives you score history without building your own. The SDK’s evaluate() takes a target function, a dataset name and a list of evaluators; an evaluator can accept inputs, outputs and reference_outputs and return a score.

# fragment: needs LANGSMITH_API_KEY and your own answer_question()
from langsmith import Client, evaluate
client = Client()
dataset = client.create_dataset("ordersportal-support-golden", description="Support questions with reference answers")
client.create_examples(
dataset_id=dataset.id,
examples=[{"inputs": {"question": "How do I change the delivery address for ORD-1042?"},
"outputs": {"answer": "Use Edit delivery on the order page before it ships."}}],
)
def mentions_reference(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
return "Edit delivery" in outputs["answer"]
results = evaluate(
lambda inputs: {"answer": answer_question(inputs["question"])["answer"]},
data="ordersportal-support-golden",
evaluators=[mentions_reference],
experiment_prefix="ci-run",
)

Exact-string evaluators like this one are cheap and deterministic; use them for facts that must appear, and keep LLM judges for qualities that need judgment.

Pull-request evals only run when your code changes. Providers update and retire models on their own schedule, so also run the full suite on a timer and alert on drift.

.github/workflows/scheduled-eval.yml
name: Weekly model health check
on:
schedule:
- cron: "0 9 * * 1" # every Monday 09:00 UTC
workflow_dispatch:
jobs:
health-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: "3.13"
- run: pip install -r evals/requirements.txt
- name: Full suite against production configuration
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
python evals/run_evals.py --dataset evals/golden_full.jsonl --output weekly.json
python evals/regression_check.py weekly.json evals/baseline.json

Pin dated model snapshots where the provider offers them, and record the exact model ID in every result so a drift alert can be traced to a model change.

Running hundreds of cases through a model and a judge on every pull request is slow and expensive. Common strategies:

Strategy How Trade-off
Tiered datasets 20-case smoke set on each PR, full 200-case set nightly and before release Faster CI, later detection of rare regressions
Concurrency asyncio.gather() with a semaphore around model calls Faster, but watch provider rate limits
Caching Reuse outputs for cases whose inputs and configuration did not change A stale cache hides real regressions; key it on prompt, model and retrieval config
Sampling Random 30% of cases per PR, full set on merge to main Probabilistic coverage
Cheaper judge Smaller judge model for smoke runs Cheaper, noisier scores; keep the strong judge for the gate that blocks releases

Sometimes a drop is the right trade-off: moving to a smaller model cuts latency by four times and costs six accuracy points the product owner is willing to give up. Record that decision instead of quietly overwriting the baseline: save a new baseline entry with approved_by and a note explaining the trade-off, in the same pull request as the change, so the history shows who accepted what and why. Never let the pipeline update its own baseline automatically after a failure; that turns the gate into a ratchet that only moves down.

1. Why do you need both an absolute threshold and a regression check? The absolute threshold protects a minimum quality level; the regression check protects the level you have already reached. A system at 0.92 faithfulness that slides to 0.80 still clears a 0.75 floor, but users will notice. Conversely, a regression check alone would accept a system that was never good enough.

2. Your eval job fails about one run in five on an unchanged branch. What do you do? Measure run-to-run variance by running the unchanged system several times, then set tolerances above that noise. Reduce noise at the source: temperature 0 for the judge, more cases per metric, majority votes for pairwise judgments. Do not allow re-runs until green; that trains the team to ignore the gate.

3. How do you keep a baseline comparison valid when the golden dataset grows? Store a hash of the dataset with each baseline and only compare runs with matching hashes. When the dataset changes, run the current production configuration on the new dataset first and approve that as the new baseline, then evaluate the candidate against it.

4. What does a scheduled evaluation catch that pull-request evaluation cannot? Changes outside your repository: a provider updating a model behind an alias, retiring a snapshot, changing safety behavior, or a data source drifting. Pull-request jobs only run when your files change.

5. Senior scenario: product wants to switch the OrdersPortal assistant to a cheaper model before the holiday peak. How do you decide? Run the full golden set on both configurations with the same judge, report per-metric deltas against the baseline and a pairwise win rate with position swapping, and add latency and cost per 1,000 requests from a benchmark run. Look at which intents regress, not only the averages; a drop on refund questions may be unacceptable even if the average is flat. If the trade-off is accepted, record a new baseline with the approver and the reason, roll out behind a flag to a share of traffic, and watch production feedback before the full switch.