Human Evaluation and Annotation

Why Human Evaluation Still Matters

Automated metrics are fast and cheap, but they have systematic blind spots. A model can score 90% on faithfulness metrics while consistently producing answers that feel awkward to real users. Human evaluation catches what automated metrics miss:

• Tone and voice — Is the response appropriately professional or friendly?
• Helpfulness — Did it actually help the user accomplish their task?
• Subtle hallucinations — Plausible-sounding wrong details that automated checkers miss
• Cultural sensitivity — Appropriate for the target audience?
• Edge case quality — How does it handle ambiguous or unusual requests?

The cost of human evaluation is higher, so the goal is to use it strategically: automate what you can, use humans for what matters most.

---

When to Use Human Evaluation

Situation	Human eval needed?
Initial model selection	Yes — automated scores alone are misleading
Prompt engineering iterations	Spot-check 10-20%
Pre-production quality gate	Yes — sample 50-100 cases
Post-launch monitoring	Yes — sample 1-5% of production traffic
A/B test between models	Mandatory for final decision
Detecting bias and fairness issues	Yes — metrics miss context
Catching subtle hallucinations	Yes — fact-checkers beat automated tools

---

The Annotation Workflow

Step 1: Sample Selection

Don’t ask evaluators to rate everything — use stratified sampling:

import random
from collections import defaultdict


def stratified_sample(
    production_logs: list[dict],
    n_samples: int = 100,
    strata_key: str = "intent_category"
) -> list[dict]:
    """Sample evenly across intent categories."""
    groups = defaultdict(list)
    for item in production_logs:
        groups[item.get(strata_key, "unknown")].append(item)

    samples_per_group = max(1, n_samples // len(groups))
    sampled = []
    for group_items in groups.values():
        sampled.extend(random.sample(group_items, min(samples_per_group, len(group_items))))

    # Top up to n_samples if groups were small
    random.shuffle(sampled)
    return sampled[:n_samples]

Step 2: Annotation Schema

Design a clear, consistent rubric before asking anyone to evaluate:

ANNOTATION_SCHEMA = {
    "overall_quality": {
        "type": "likert",
        "scale": [1, 2, 3, 4, 5],
        "labels": {
            1: "Completely wrong or unhelpful",
            2: "Partially correct but significantly flawed",
            3: "Acceptable but room for improvement",
            4: "Good — meets the user's need",
            5: "Excellent — clearly helpful and accurate",
        },
    },
    "factual_accuracy": {
        "type": "categorical",
        "options": ["accurate", "minor_error", "major_error", "cannot_assess"],
    },
    "tone_appropriateness": {
        "type": "binary",
        "options": ["appropriate", "inappropriate"],
    },
    "helpful": {
        "type": "binary",
        "options": ["yes", "no"],
    },
    "free_text_feedback": {
        "type": "text",
        "required": False,
    },
}

Step 3: Build an Annotation Interface

A simple annotation tool using Streamlit:

annotate.py

import json
import streamlit as st
from pathlib import Path

samples = json.loads(Path("samples_to_annotate.json").read_text())
results = []

st.title("AI Output Annotation")

for i, sample in enumerate(samples):
    st.divider()
    st.subheader(f"Sample {i + 1} of {len(samples)}")

    col1, col2 = st.columns(2)
    with col1:
        st.markdown("**User Question:**")
        st.info(sample["question"])
    with col2:
        st.markdown("**AI Response:**")
        st.success(sample["ai_response"])

    quality = st.slider("Overall Quality", 1, 5, 3, key=f"quality_{i}")
    accurate = st.radio("Factually accurate?", ["Accurate", "Minor error", "Major error"], key=f"acc_{i}")
    helpful = st.checkbox("Was this actually helpful?", key=f"help_{i}")
    notes = st.text_input("Notes (optional):", key=f"notes_{i}")

    results.append({
        "sample_id": sample["id"],
        "quality": quality,
        "accuracy": accurate,
        "helpful": helpful,
        "notes": notes,
    })

if st.button("Save Annotations"):
    Path("annotations_output.json").write_text(json.dumps(results, indent=2))
    st.success("Saved!")

streamlit run annotate.py

---

Inter-Annotator Agreement

When multiple people evaluate the same samples, you need to measure how much they agree — disagreement signals an ambiguous rubric.

from sklearn.metrics import cohen_kappa_score
import numpy as np


def compute_agreement(annotations_a: list[int], annotations_b: list[int]) -> dict:
    """Compute agreement metrics between two annotators."""

    # Cohen's Kappa — accounts for chance agreement
    kappa = cohen_kappa_score(annotations_a, annotations_b)

    # Raw agreement rate
    agreement_rate = sum(a == b for a, b in zip(annotations_a, annotations_b)) / len(annotations_a)

    # Within-1 agreement (useful for Likert scales)
    within_one = sum(abs(a - b) <= 1 for a, b in zip(annotations_a, annotations_b)) / len(annotations_a)

    interpretation = (
        "excellent" if kappa > 0.8 else
        "good" if kappa > 0.6 else
        "moderate" if kappa > 0.4 else
        "poor — rubric needs clarification"
    )

    return {
        "cohen_kappa": round(kappa, 3),
        "raw_agreement": round(agreement_rate, 3),
        "within_one_agreement": round(within_one, 3),
        "interpretation": interpretation,
    }


# Example
annotator_a = [4, 3, 5, 2, 4, 3, 5, 4, 2, 3]
annotator_b = [4, 3, 5, 3, 4, 2, 5, 4, 2, 3]
print(compute_agreement(annotator_a, annotator_b))
# {'cohen_kappa': 0.847, 'raw_agreement': 0.9, 'within_one_agreement': 1.0, 'interpretation': 'excellent'}

Kappa benchmarks:

• 0.8: Near-perfect agreement — your rubric is clear

• 0.6–0.8: Good agreement — minor clarifications needed
• 0.4–0.6: Moderate — rubric is ambiguous, discuss examples
• < 0.4: Poor — you need calibration sessions before continuing

---

LLM-as-Judge

Using a frontier model (GPT-4o or Claude Sonnet) as an evaluator can scale human-quality judgment at low cost. It’s not as reliable as human judgment for edge cases but is far faster.

import anthropic

client = anthropic.Anthropic()


def llm_judge(
    question: str,
    response: str,
    reference_answer: str,
    rubric: str = "accuracy and helpfulness",
) -> dict:
    """Use Claude as a judge to score a response."""

    result = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": f"""Evaluate this AI response based on: {rubric}

Question: {question}

Reference answer: {reference_answer}

AI response: {response}

Score 1-5 (1=poor, 5=excellent) and explain in one sentence.
Format: SCORE: X\nREASON: ..."""
        }]
    )

    text = result.content[0].text
    lines = text.strip().split("\n")
    score_line = next((l for l in lines if l.startswith("SCORE:")), "SCORE: 3")
    reason_line = next((l for l in lines if l.startswith("REASON:")), "REASON: No reason provided")

    return {
        "score": int(score_line.split(":")[1].strip()),
        "reason": reason_line.split(":", 1)[1].strip(),
    }

LLM judge calibration — compare LLM scores against human scores on 50 examples to measure how well the judge correlates with human judgment before relying on it:

def calibrate_llm_judge(human_scores: list[int], llm_scores: list[int]) -> float:
    """Compute Pearson correlation between human and LLM judge scores."""
    from scipy.stats import pearsonr
    correlation, p_value = pearsonr(human_scores, llm_scores)
    print(f"LLM-Human correlation: r={correlation:.3f} (p={p_value:.4f})")
    return correlation

---

Production Feedback Loops

Collect real user feedback on AI responses in production:

feedback_collection.py

from fastapi import FastAPI
from pydantic import BaseModel
from datetime import datetime
import json
from pathlib import Path

app = FastAPI()
FEEDBACK_LOG = Path("production_feedback.jsonl")


class Feedback(BaseModel):
    conversation_id: str
    message_id: str
    rating: int          # 1-5
    feedback_text: str = ""
    was_helpful: bool


@app.post("/feedback")
async def submit_feedback(feedback: Feedback):
    entry = {
        "timestamp": datetime.utcnow().isoformat(),
        **feedback.model_dump(),
    }
    with FEEDBACK_LOG.open("a") as f:
        f.write(json.dumps(entry) + "\n")
    return {"status": "recorded"}

Route negative feedback to human review:

def process_feedback_queue():
    """Surface low-rated responses for human review."""
    feedback = [json.loads(l) for l in FEEDBACK_LOG.read_text().strip().splitlines()]

    low_rated = [f for f in feedback if f["rating"] <= 2]

    if len(low_rated) > 0:
        daily_rate = len(low_rated) / len(feedback)
        print(f"Low-rated responses: {len(low_rated)} ({daily_rate:.1%})")

        if daily_rate > 0.10:  # Alert if >10% get low ratings
            send_alert(f"High negative feedback rate: {daily_rate:.1%}")

    return low_rated

---

Building the Annotation → Training Feedback Loop

Human annotations on production failures can directly improve your system:

Production Response (low quality)
    │
    ▼
Human Annotator reviews + corrects
    │
    ▼
Correction stored as new golden example
    │
    ▼
Golden dataset grows over time
    │
    ▼
Automated eval harness runs against growing dataset
    │
    ▼
Model or prompt improvements measured against real failures

This is the virtuous cycle that separates continuously improving AI systems from ones that plateau after launch.