Skip to content

OpenAI Codex (2021): the original code model

  • What the original Codex model was, when it existed, and why the name now means something else.
  • How it was built: a GPT language model fine-tuned on public source code.
  • How OpenAI measured it, with the HumanEval benchmark and the pass@k metric, and how to compute pass@k yourself.
  • The limitations the Codex paper itself documented, and which of them still apply to today’s coding models.
  • How Codex connects to GitHub Copilot and to the coding agents that followed.

In July 2021 OpenAI published “Evaluating Large Language Models Trained on Code” (Chen et al.), which introduced Codex: GPT language models fine-tuned on publicly available code from GitHub. The paper reported that a 12-billion-parameter Codex model solved 28.8% of a new benchmark of hand-written Python problems, HumanEval, with a single sample, where GPT-3 solved 0% and GPT-J 11.4%. Sampling 100 completions per problem and checking each against unit tests raised that to 70.2%. The paper also stated that a distinct production version of Codex powered GitHub Copilot.

OpenAI then offered Codex models in its API as a free limited beta. They were deprecated with three days’ notice on March 20, 2023 and shut down on March 23, 2023 (OpenAI’s deprecations page lists code-davinci-002 among them). General GPT models had absorbed their capability.

Codex matters today for three reasons: it showed that a general language model plus code data could write working programs, it gave the field HumanEval and pass@k, and its paper named most of the risks that teams still manage when they adopt coding assistants.

Date Event
July 2021 Codex paper published; HumanEval and the pass@k estimator released in the openai/human-eval repository
2021 Codex models available in the OpenAI API as a free limited beta; a production version powers GitHub Copilot
March 20, 2023 OpenAI announces the Codex model shutdown
March 23, 2023 Codex models shut down in the API; OpenAI’s deprecations page now lists gpt-4o as the replacement
2025 onward OpenAI reuses the name Codex for a coding agent (cloud tasks and a CLI), unrelated to the old API model

Codex was a decoder-only GPT model, the same architecture as GPT-3, trained further on code. The paper’s main corpus was 159 GB of Python files collected from public GitHub repositories after filtering out likely auto-generated files and files with very long lines. The authors tried starting from a pre-trained GPT-3 checkpoint and from scratch; starting from GPT-3 did not raise final quality on this large dataset, but it converged faster, so they used it.

Usage was completion, not chat. You gave the model the start of a file, typically imports, a function signature and a docstring, and it continued the text. Everything it “knew” about your intent came from that prompt, which is why docstrings, type hints and examples in comments changed results so much. The paper also trained Codex-S, fine-tuned further on correctly implemented standalone functions, which reached 37.7% on HumanEval with one sample.

Two consequences of that design shaped every early coding tool:

  • No execution, no feedback. The model never ran its code. Correctness came from whatever checked the output afterwards: a test suite, a compiler, or a developer.
  • Context was a window of text. Codex saw only the prompt. Tools such as Copilot had to choose which surrounding code to include, and that selection determined quality as much as the model did.

How Codex was evaluated: HumanEval and pass@k

Section titled “How Codex was evaluated: HumanEval and pass@k”

Earlier code-generation work scored outputs by text similarity to a reference solution. The Codex paper argued that this misses the point: two very different programs can both be correct, and a near-identical one can be wrong. It measured functional correctness instead.

HumanEval is 164 hand-written Python problems, each with a function signature, a docstring, a canonical solution and unit tests. (The released dataset file contains exactly 164 tasks.) A completion counts as correct only if it passes the problem’s tests.

pass@k is the probability that at least one of k sampled completions passes. Estimating it naively from k samples has high variance, so the paper generates n ≥ k samples per problem, counts the c that pass, and uses an unbiased estimator: 1 − C(n − c, k) / C(n, k), averaged over problems. The human-eval repository implements a numerically stable form of the same formula.

pass@1 approximates “the first answer is right”. pass@100 measures “a correct answer exists among many”, which is only useful if something can pick it out, for example a test suite. The paper reported that choosing the sample with the highest mean token log-probability recovered much of the gap between pass@1 and pass@100 when tests were not available.

The sample needs only Python 3 (tested with Python 3.11). It uses hard-coded strings in place of model output so it runs offline and deterministically, and it execs them only because they are written in the file; the HumanEval repository warns that real model output must run in a sandbox. Save it as pass_at_k.py and run python3 pass_at_k.py.

An invented shop, OrdersPortal, wants a helper that totals an order in cents and applies a whole-number percentage discount, rounding down. The prompt is the signature and docstring, in Codex style. Five “sampled completions” are graded against unit tests, then pass@k is computed with the Codex estimator.

# pass_at_k.py - grade sampled completions the way the 2021 Codex paper did (HumanEval-style)
# The "samples" are hard-coded strings standing in for model output, so this runs offline.
from math import comb
PROMPT = '''
def order_total(lines, discount_pct=0):
"""Return the OrdersPortal order total in cents.
lines: list of (unit_price_cents, quantity). Discount is a whole-number percent,
applied to the subtotal and rounded down to the nearest cent."""
'''
SAMPLES = [ # five completions a code model might produce for the same prompt
" s = sum(p * q for p, q in lines)\n return s - s * discount_pct // 100\n",
" s = sum(p * q for p, q in lines)\n return int(s * (1 - discount_pct / 100))\n",
" return sum(p for p, q in lines) * (100 - discount_pct) // 100\n",
" s = sum(p * q for p, q in lines)\n return s * (100 - discount_pct) // 100\n",
" s = 0\n for p, q in lines:\n s += p * q\n return round(s * (100 - discount_pct) / 100)\n",
]
def weak_tests(fn): # the four tests a quick reviewer might write
assert fn([(1999, 2), (500, 1)]) == 4498
assert fn([(1999, 2), (500, 1)], 10) == 4048 # 4498 - 449.8, rounded down
assert fn([(333, 3)], 15) == 849 # 999 * 0.85 = 849.15
assert fn([], 50) == 0
def strong_tests(fn): # the same four plus two edge cases
weak_tests(fn)
assert fn([(1000, 1)], 7) == 930 # 930.0 exactly; float math gives 929
assert fn([(3, 1)], 50) == 1 # 1.5 rounds DOWN to 1, not to 2
def passes(completion: str, tests) -> bool:
scope: dict = {}
try:
exec(PROMPT + completion, scope) # only safe because we wrote the samples
tests(scope["order_total"])
return True
except Exception:
return False
def pass_at_k(n: int, c: int, k: int) -> float:
"""Unbiased estimator from the Codex paper: 1 - C(n-c, k) / C(n, k)."""
return 1.0 if n - c < k else 1.0 - comb(n - c, k) / comb(n, k)
for name, tests in (("weak tests", weak_tests), ("strong tests", strong_tests)):
results = [passes(s, tests) for s in SAMPLES]
n, c = len(results), sum(results)
print(f"{name}:", ["pass" if r else "fail" for r in results])
print(" " + " ".join(f"pass@{k} = {pass_at_k(n, c, k):.3f}" for k in (1, 2, 5)))

Output:

weak tests: ['fail', 'pass', 'fail', 'pass', 'pass']
pass@1 = 0.600 pass@2 = 0.900 pass@5 = 1.000
strong tests: ['fail', 'fail', 'fail', 'pass', 'fail']
pass@1 = 0.200 pass@2 = 0.400 pass@5 = 1.000

What the example shows:

  • Plausible code fails in small ways. Sample 1 subtracts a rounded-down discount, so the total rounds up by one cent. Sample 3 ignores quantity. Both look reasonable at a glance.
  • Passing tests is not the same as correct. The four weak tests accept three samples, but only sample 4 is actually correct. Sample 2 uses floating point and returns 929 instead of 930 for a 1,000-cent order at 7%; sample 5 uses round(), so a 3-cent line at 50% returns 2 instead of 1. Two extra edge-case tests drop the measured pass@1 from 0.6 to its true value, 0.2. A model’s score is only as good as the tests behind it, which is still the central problem of evaluating code models.
  • pass@k rewards a selector. With strong tests, one sample in five is correct, yet pass@5 is still 1.0. That number is only real if something in your workflow, usually a test suite strong enough to reject samples 2 and 5, can identify the correct sample.

The Codex paper was unusually frank about risks, and most of its list still applies to modern coding models:

  • Misalignment with the user’s intent. Given a prompt containing subtle bugs, Codex tended to produce code that continued the bugs. It optimized for “likely next text”, not “what the developer meant”.
  • Degradation with longer specifications. Performance dropped as docstrings chained more operations together.
  • Insecure or outdated patterns. Trained on public code, it reproduced common mistakes, including insecure configurations and deprecated APIs.
  • Over-reliance. The authors flagged that developers, especially novices, might accept suggestions without understanding them.
  • Licensing and attribution questions about generated code that resembles training data.
  • Bias inherited from code and comments in the training data.

Codex’s capabilities moved into OpenAI’s general GPT models, which is why the separate code models could be shut down in 2023. Coding tools then changed shape: from completing the current line, to chat about code, to agents that read a repository, edit files, run tests and iterate. GitHub Copilot now lets users choose among models from several providers; see GitHub Copilot.

In 2025 OpenAI reused the Codex name for a coding agent that runs tasks in cloud sandboxes and through a local CLI. It shares nothing with the 2021 model except the name and the problem it addresses. If an article or job description says “Codex”, check which one it means: a mention of code-davinci-002, completions or HumanEval is about the old model; a mention of tasks, pull requests or a CLI is about the agent.

  • “Codex is still an API model.” It has not been callable since March 23, 2023. Code that names code-davinci-002 fails.
  • “Codex was GPT-3 with no further training.” It was fine-tuned on a large code corpus, and the paper shows the fine-tuning, not the base model, produced the gains.
  • “High pass@100 means the model is usually right.” It means a correct answer usually exists among many samples. Without a reliable selector, pass@1 is the relevant number.
  • “The Codex agent is the same model, updated.” It is a separate product built on current GPT models.

1. What was the original OpenAI Codex, and why is it no longer available? A GPT model fine-tuned on public code, published in 2021, offered in the API as a limited beta and in production behind the first GitHub Copilot. OpenAI shut the Codex API models down on March 23, 2023, because its general GPT models had matched and surpassed them.

2. What is pass@k, and why does the Codex paper use an unbiased estimator? pass@k is the probability that at least one of k samples passes the unit tests. Computing it from exactly k samples per problem is noisy, so the paper draws n ≥ k samples, counts c correct, and computes 1 − C(n − c, k) / C(n, k) per problem, then averages.

3. Why did the paper prefer functional correctness over text-match metrics? Because code that matches a reference closely can be wrong and code that looks different can be right. Running unit tests measures what matters, behavior, and it lets the benchmark accept any correct solution.

4. Senior scenario: your team reports that an AI coding tool “scores 90% on our benchmark”, and leadership wants to make it mandatory for all OrdersPortal services. Using what the Codex work taught, how do you assess the claim? Ask which metric: pass@1 or pass@k with large k, and whether the workflow has a reliable selector such as tests that runs before code is accepted. Ask how strong the benchmark’s tests are: the worked example shows a floating-point bug passing four tests, so a weak test suite inflates scores. Check for contamination, since public benchmarks may be in the training data; build a private set from the team’s own recent tickets. Measure in the real workflow, not in isolation: review time, escaped defects, security findings and rework, comparing teams with and without the tool. Roll out with guardrails the Codex paper already called for: mandatory code review, secret and dependency scanning, license checks on large generated blocks, and training so juniors do not accept code they cannot explain. Make it the default only after those measurements, and keep a way to opt out for sensitive code paths.

  • The original Codex (2021 to 2023) was a GPT model fine-tuned on code; its API models were shut down on March 23, 2023.
  • Its paper introduced HumanEval (164 Python problems with tests) and the unbiased pass@k estimator, still used to evaluate code models.
  • Functional correctness is only as good as the tests; pass@k is only useful with a way to select the right sample.
  • The risks it documented, from following buggy prompts to insecure patterns and over-reliance, still apply.
  • Today’s “Codex” is a different product: an OpenAI coding agent.