LLM Evaluation Frameworks

Overview

Evaluation frameworks provide the infrastructure to run benchmarks: they load datasets, call models, score responses, and report results. Choosing the right framework depends on whether you’re evaluating base models, fine-tuned models, RAG pipelines, or production prompts.

lm-evaluation-harness (EleutherAI)

The most widely used open-source harness for benchmarking base LLMs. Used by researchers to produce the scores you see on leaderboards like Open LLM Leaderboard.

Best for: Comparing base models against standard academic benchmarks.

pip install lm-eval

# Evaluate Llama 3.2 on MMLU and HellaSwag
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.2-3B \
  --tasks mmlu,hellaswag \
  --device cuda:0 \
  --batch_size 8 \
  --output_path ./results

Built-in tasks include:

mmlu — 57-subject knowledge benchmark
hellaswag — commonsense reasoning
truthfulqa — measures tendency to hallucinate
gsm8k — grade school math
humaneval — Python code generation

# Programmatic API
import lm_eval

results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=meta-llama/Llama-3.2-3B",
    tasks=["mmlu", "gsm8k"],
    batch_size=8
)

print(results["results"])
# {'mmlu': {'acc,none': 0.612, ...}, 'gsm8k': {'exact_match,strict-match': 0.541, ...}}

When to use: You’re selecting a base model to fine-tune, or need to reproduce scores from a research paper.

HELM (Stanford CRFM)

Holistic Evaluation of Language Models — evaluates models across multiple dimensions simultaneously: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.

Best for: Comprehensive model auditing before procurement or deployment.

pip install crfm-helm

# Run a HELM scenario
helm-run --conf-paths run_specs.conf \
  --output-path benchmark_output \
  --suite my-eval-run

helm-summarize --suite my-eval-run

# run_specs.conf — example config
entries:
  - description: "mmlu:subject=abstract_algebra,model=openai/gpt-4o"
    priority: 1
  - description: "boolq:model=openai/gpt-4o"
    priority: 1

HELM produces a radar chart across all dimensions, making it easy to see if a model is accurate but biased, or fast but poorly calibrated.

OpenAI Evals

A framework for evaluating OpenAI models (and others via the API) against custom task definitions. OpenAI uses it internally and has open-sourced the tooling.

Best for: Product teams building on the OpenAI API who need custom evaluation datasets.

pip install evals

# Run a built-in eval
oaieval gpt-4o test-basic

Writing a custom eval:

my-qa-eval:
  id: my-qa-eval.v1
  metrics: [accuracy]
  description: Custom Q&A evaluation for our support bot

my-qa-eval.v1:
  class: evals.elsuite.basic.match:Match
  args:
    samples_jsonl: my-qa-eval/samples.jsonl

{"input": [{"role": "user", "content": "What is our refund policy?"}], "ideal": "30 days"}
{"input": [{"role": "user", "content": "How do I reset my password?"}], "ideal": "Use the forgot password link on the login page"}

oaieval gpt-4o my-qa-eval

DeepEval

A developer-friendly framework modelled on pytest. Write eval test cases the same way you write unit tests.

Best for: Engineering teams who want LLM evaluation to feel like standard testing.

pip install deepeval

import pytest
from deepeval import assert_test
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
    ToxicityMetric
)
from deepeval.test_case import LLMTestCase

@pytest.mark.parametrize("test_case", [
    LLMTestCase(
        input="How do I cancel my subscription?",
        actual_output="Go to Settings > Billing > Cancel Subscription.",
        retrieval_context=["Cancellation can be done via Settings > Billing."]
    ),
    LLMTestCase(
        input="What is your pricing?",
        actual_output="We charge $29/month for the Pro plan.",
        retrieval_context=["Pro plan costs $29 per month."]
    )
])
def test_support_bot(test_case: LLMTestCase):
    assert_test(test_case, [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.8),
        HallucinationMetric(threshold=0.3),
    ])

deepeval test run test_support_bot.py

# Output:
# ✓ test_support_bot[test_case0] PASSED
# ✓ test_support_bot[test_case1] PASSED

DeepEval metrics:

AnswerRelevancyMetric — is the answer relevant to the question?
FaithfulnessMetric — does the answer stay true to the retrieved context?
HallucinationMetric — does the answer contradict known facts?
GEval — custom criteria using LLM-as-judge
BiasMetric — detects demographic bias
ToxicityMetric — flags harmful content

Promptfoo

Designed specifically for comparing prompts and models side-by-side. Think A/B testing for prompts.

Best for: Prompt engineers who want to compare prompt variations or models against each other.

npm install -g promptfoo
promptfoo init

prompts:
  - "Answer concisely: {{question}}"
  - "You are a helpful assistant. Answer in 1-2 sentences: {{question}}"

providers:
  - openai:gpt-4o
  - anthropic:claude-sonnet-4-6

tests:
  - vars:
      question: "What is the capital of France?"
    assert:
      - type: contains
        value: "Paris"
      - type: llm-rubric
        value: "Is factually correct and concise"

  - vars:
      question: "Explain recursion to a 10-year-old"
    assert:
      - type: llm-rubric
        value: "Uses simple language appropriate for a child"
      - type: javascript
        value: output.length < 500

promptfoo eval
promptfoo view  # Opens interactive comparison UI

Comparison Summary

Framework	Best for	Model support	Custom tasks	UI
lm-evaluation-harness	Base model benchmarking	HF, API	Yes	No
HELM	Holistic auditing	API	Limited	Web
OpenAI Evals	OpenAI-specific tasks	OpenAI API	Yes	No
DeepEval	pytest-style testing	Any (via adapter)	Yes	Dashboard
Promptfoo	Prompt A/B testing	Any	Yes	Yes
Ragas	RAG evaluation	Any	Yes	No
LangSmith	LangChain tracing + eval	LangChain models	Yes	Yes

Which Should You Use?

Start with DeepEval or Promptfoo if you’re a product team that needs evaluation quickly and wants it to integrate naturally with your codebase and CI.

Use lm-evaluation-harness if you’re selecting or comparing base models for fine-tuning and need reproducible benchmark scores.

Use HELM if you need a comprehensive audit covering fairness, toxicity, and robustness — for compliance or procurement decisions.

Use Ragas if your application is a RAG system — it was built specifically for that task.