LLM Evaluation Frameworks
Overview
Section titled “Overview”Evaluation frameworks provide the infrastructure to run benchmarks: they load datasets, call models, score responses, and report results. Choosing the right framework depends on whether you’re evaluating base models, fine-tuned models, RAG pipelines, or production prompts.
lm-evaluation-harness (EleutherAI)
Section titled “lm-evaluation-harness (EleutherAI)”The most widely used open-source harness for benchmarking base LLMs. Used by researchers to produce the scores you see on leaderboards like Open LLM Leaderboard.
Best for: Comparing base models against standard academic benchmarks.
pip install lm-eval
# Evaluate Llama 3.2 on MMLU and HellaSwaglm_eval --model hf \ --model_args pretrained=meta-llama/Llama-3.2-3B \ --tasks mmlu,hellaswag \ --device cuda:0 \ --batch_size 8 \ --output_path ./resultsBuilt-in tasks include:
mmlu— 57-subject knowledge benchmarkhellaswag— commonsense reasoningtruthfulqa— measures tendency to hallucinategsm8k— grade school mathhumaneval— Python code generation
# Programmatic APIimport lm_eval
results = lm_eval.simple_evaluate( model="hf", model_args="pretrained=meta-llama/Llama-3.2-3B", tasks=["mmlu", "gsm8k"], batch_size=8)
print(results["results"])# {'mmlu': {'acc,none': 0.612, ...}, 'gsm8k': {'exact_match,strict-match': 0.541, ...}}When to use: You’re selecting a base model to fine-tune, or need to reproduce scores from a research paper.
HELM (Stanford CRFM)
Section titled “HELM (Stanford CRFM)”Holistic Evaluation of Language Models — evaluates models across multiple dimensions simultaneously: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
Best for: Comprehensive model auditing before procurement or deployment.
pip install crfm-helm
# Run a HELM scenariohelm-run --conf-paths run_specs.conf \ --output-path benchmark_output \ --suite my-eval-run
helm-summarize --suite my-eval-run# run_specs.conf — example configentries: - description: "mmlu:subject=abstract_algebra,model=openai/gpt-6.1-terra" priority: 1 - description: "boolq:model=openai/gpt-6.1-terra" priority: 1HELM produces a radar chart across all dimensions, making it easy to see if a model is accurate but biased, or fast but poorly calibrated.
OpenAI Evals
Section titled “OpenAI Evals”A framework for evaluating OpenAI models (and others via the API) against custom task definitions. OpenAI uses it internally and has open-sourced the tooling.
Best for: Product teams building on the OpenAI API who need custom evaluation datasets.
pip install evals
# Run a built-in evaloaieval gpt-6.1-terra test-basicWriting a custom eval:
my-qa-eval: id: my-qa-eval.v1 metrics: [accuracy] description: Custom Q&A evaluation for our support bot
my-qa-eval.v1: class: evals.elsuite.basic.match:Match args: samples_jsonl: my-qa-eval/samples.jsonl{"input": [{"role": "user", "content": "What is our refund policy?"}], "ideal": "30 days"}{"input": [{"role": "user", "content": "How do I reset my password?"}], "ideal": "Use the forgot password link on the login page"}oaieval gpt-6.1-terra my-qa-evalDeepEval
Section titled “DeepEval”A developer-friendly framework modelled on pytest. Write eval test cases the same way you write unit tests.
Best for: Engineering teams who want LLM evaluation to feel like standard testing.
pip install deepevalimport pytestfrom deepeval import assert_testfrom deepeval.metrics import ( AnswerRelevancyMetric, FaithfulnessMetric, HallucinationMetric, ToxicityMetric)from deepeval.test_case import LLMTestCase
@pytest.mark.parametrize("test_case", [ LLMTestCase( input="How do I cancel my subscription?", actual_output="Go to Settings > Billing > Cancel Subscription.", retrieval_context=["Cancellation can be done via Settings > Billing."] ), LLMTestCase( input="What is your pricing?", actual_output="We charge $29/month for the Pro plan.", retrieval_context=["Pro plan costs $29 per month."] )])def test_support_bot(test_case: LLMTestCase): assert_test(test_case, [ AnswerRelevancyMetric(threshold=0.7), FaithfulnessMetric(threshold=0.8), HallucinationMetric(threshold=0.3), ])deepeval test run test_support_bot.py
# Output:# ✓ test_support_bot[test_case0] PASSED# ✓ test_support_bot[test_case1] PASSEDDeepEval metrics:
AnswerRelevancyMetric— is the answer relevant to the question?FaithfulnessMetric— does the answer stay true to the retrieved context?HallucinationMetric— does the answer contradict known facts?GEval— custom criteria using LLM-as-judgeBiasMetric— detects demographic biasToxicityMetric— flags harmful content
Promptfoo
Section titled “Promptfoo”Designed specifically for comparing prompts and models side-by-side. Think A/B testing for prompts.
Best for: Prompt engineers who want to compare prompt variations or models against each other.
npm install -g promptfoopromptfoo initprompts: - "Answer concisely: {{question}}" - "You are a helpful assistant. Answer in 1-2 sentences: {{question}}"
providers: - openai:gpt-6.1-terra - anthropic:claude-sonnet-5-5
tests: - vars: question: "What is the capital of France?" assert: - type: contains value: "Paris" - type: llm-rubric value: "Is factually correct and concise"
- vars: question: "Explain recursion to a 10-year-old" assert: - type: llm-rubric value: "Uses simple language appropriate for a child" - type: javascript value: output.length < 500promptfoo evalpromptfoo view # Opens interactive comparison UIComparison Summary
Section titled “Comparison Summary”| Framework | Best for | Model support | Custom tasks | UI |
|---|---|---|---|---|
| lm-evaluation-harness | Base model benchmarking | HF, API | Yes | No |
| HELM | Holistic auditing | API | Limited | Web |
| OpenAI Evals | OpenAI-specific tasks | OpenAI API | Yes | No |
| DeepEval | pytest-style testing | Any (via adapter) | Yes | Dashboard |
| Promptfoo | Prompt A/B testing | Any | Yes | Yes |
| Ragas | RAG evaluation | Any | Yes | No |
| LangSmith | LangChain tracing + eval | LangChain models | Yes | Yes |
Which Should You Use?
Section titled “Which Should You Use?”Start with DeepEval or Promptfoo if you’re a product team that needs evaluation quickly and wants it to integrate naturally with your codebase and CI.
Use lm-evaluation-harness if you’re selecting or comparing base models for fine-tuning and need reproducible benchmark scores.
Use HELM if you need a comprehensive audit covering fairness, toxicity, and robustness — for compliance or procurement decisions.
Use Ragas if your application is a RAG system — it was built specifically for that task.