Evaluating RAG Systems

Why RAG Evaluation is Different

A standard LLM evaluation asks: did the model answer the question correctly? A RAG evaluation has two layers:

Retrieval quality — did the system fetch the right documents?
Generation quality — did the model use those documents to produce a correct answer?

A failure can happen at either layer. Evaluating only the final answer doesn’t tell you which part broke.

User Query
    │
    ▼
[Retriever] ──► Retrieved Chunks  ◄── Evaluate: Is this relevant?
    │
    ▼
[LLM Generator] ──► Final Answer  ◄── Evaluate: Is this faithful to the chunks?
                                       Evaluate: Does this answer the question?

Ragas — RAG-Specific Evaluation Framework

Ragas was built specifically to evaluate RAG pipelines. It provides four core metrics that map directly to the two-layer RAG problem.

pip install ragas

The Four Core Ragas Metrics

Metric	What it measures	Score range
Faithfulness	Does the answer stick to the retrieved context? (no hallucinations)	0–1
Answer Relevancy	Does the answer actually address the question?	0–1
Context Recall	Did retrieval fetch all the information needed?	0–1
Context Precision	Are the retrieved chunks actually useful (not noisy)?	0–1

Running Ragas

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
)

# Your RAG test data
data = {
    "question": [
        "What is the refund policy?",
        "How do I upgrade my account?",
        "What payment methods do you accept?",
    ],
    "answer": [
        "You can request a refund within 30 days of purchase.",
        "Go to Settings > Billing > Upgrade Plan.",
        "We accept Visa, Mastercard, and PayPal.",
    ],
    "contexts": [
        # What the retriever actually fetched for each question
        [["Our refund policy allows returns within 30 days."]],
        [["Account upgrades can be done in the Billing section of Settings."]],
        [["Accepted payment methods: Visa, Mastercard, PayPal, and Apple Pay."]],
    ],
    "ground_truth": [
        # The ideal answer (for context_recall)
        "Refunds are available within 30 days.",
        "Upgrade via Settings > Billing.",
        "We accept Visa, Mastercard, PayPal, and Apple Pay.",
    ],
}

dataset = Dataset.from_dict(data)

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
)

print(results)
# {'faithfulness': 0.95, 'answer_relevancy': 0.88,
#  'context_recall': 0.73, 'context_precision': 0.82}

Interpreting the Results

Faithfulness 0.95 — almost all answers are grounded in retrieved context. Good.
Answer Relevancy 0.88 — most answers address the question. Acceptable.
Context Recall 0.73 — retriever missed relevant information 27% of the time. Needs improvement.
Context Precision 0.82 — 18% of retrieved chunks were irrelevant noise. Consider re-ranking.

Low context recall → improve your chunking strategy or increase k (number of retrieved chunks).
Low context precision → improve your re-ranking model or reduce chunk size.

Building a Full RAG Evaluation Pipeline

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.chains import RetrievalQA
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

# Your RAG pipeline
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.load_local("./my_vectorstore", embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
llm = ChatOpenAI(model="gpt-4o")
rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

# Load your golden test set
import json
test_cases = [json.loads(l) for l in open("golden_dataset.jsonl")]

# Run each test case through your RAG pipeline
results_data = {
    "question": [],
    "answer": [],
    "contexts": [],
    "ground_truth": [],
}

for case in test_cases:
    result = rag_chain.invoke({"query": case["input"]})

    results_data["question"].append(case["input"])
    results_data["answer"].append(result["result"])
    results_data["contexts"].append(
        [doc.page_content for doc in result["source_documents"]]
    )
    results_data["ground_truth"].append(case["expected_output"])

# Evaluate
dataset = Dataset.from_dict(results_data)
scores = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_recall])
print(scores)

TruLens — Feedback Functions for LLM Apps

TruLens provides a broader evaluation approach using feedback functions — any function that takes LLM inputs/outputs and returns a quality score.

pip install trulens-eval

from trulens_eval import Tru, TruChain, Feedback
from trulens_eval.feedback.provider import OpenAI as TruOpenAI

tru = Tru()
provider = TruOpenAI()

# Define feedback functions
f_answer_relevance = Feedback(
    provider.relevance_with_cot_reasons,
    name="Answer Relevance"
).on_input_output()

f_context_relevance = Feedback(
    provider.context_relevance_with_cot_reasons,
    name="Context Relevance"
).on_input().on(TruChain.select_context()).aggregate(np.mean)

f_groundedness = Feedback(
    provider.groundedness_measure_with_cot_reasons,
    name="Groundedness"
).on(TruChain.select_context()).on_output().aggregate(np.mean)

# Wrap your LangChain chain
tru_rag = TruChain(
    rag_chain,
    app_id="my-rag-app",
    feedbacks=[f_answer_relevance, f_context_relevance, f_groundedness]
)

# Run evaluation
with tru_rag as recording:
    for case in test_cases:
        rag_chain.invoke({"query": case["input"]})

# View results in the TruLens dashboard
tru.run_dashboard()

Custom Retrieval Metrics

Sometimes you need metrics beyond what libraries provide. Two important custom metrics:

Mean Reciprocal Rank (MRR)

How quickly does the retriever surface the right document?

def mean_reciprocal_rank(ranked_docs: list[str], relevant_doc_ids: set[str]) -> float:
    """MRR: 1/rank of first relevant document. Higher is better."""
    for rank, doc_id in enumerate(ranked_docs, start=1):
        if doc_id in relevant_doc_ids:
            return 1.0 / rank
    return 0.0  # no relevant document found

# Example
ranked = ["doc_5", "doc_1", "doc_3"]
relevant = {"doc_1", "doc_7"}
mrr = mean_reciprocal_rank(ranked, relevant)
# 1/2 = 0.5 — relevant doc was at rank 2

Hit Rate @ K

What fraction of queries had at least one relevant document in the top K results?

def hit_rate_at_k(test_cases: list[dict], retriever, k: int = 5) -> float:
    hits = 0
    for case in test_cases:
        retrieved = retriever.get_relevant_documents(case["query"])[:k]
        retrieved_ids = {doc.metadata["doc_id"] for doc in retrieved}
        if case["relevant_doc_id"] in retrieved_ids:
            hits += 1
    return hits / len(test_cases)

RAG Evaluation Dashboard

Track scores over time as you iterate on your pipeline:

import json
from datetime import datetime
from pathlib import Path

def log_rag_scores(scores: dict, version: str):
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "version": version,
        "scores": scores,
    }
    log_file = Path("rag_eval_history.jsonl")
    with log_file.open("a") as f:
        f.write(json.dumps(log_entry) + "\n")

# After each evaluation run
log_rag_scores(
    scores={"faithfulness": 0.95, "answer_relevancy": 0.88, "context_recall": 0.73},
    version="v2.3-chunk-512"
)

Plot the history to see which changes improved or hurt your pipeline.

Common RAG Failure Modes

Symptom	Likely cause	Fix
Low faithfulness	LLM ignoring retrieved context	Stronger system prompt, less creative temperature
Low context recall	Retriever missing key documents	Increase k, improve chunking, add hybrid search
Low context precision	Retriever fetching irrelevant noise	Add cross-encoder re-ranker, reduce chunk size
Low answer relevancy	Answer drifts off topic	Tighter prompt instructions
High latency	Too many retrieved chunks	Reduce k, use re-ranker to filter early