Skip to content

Python Generators: yield, send, and Lazy Evaluation

  • What a generator function is and how yield suspends and resumes execution.
  • The generator lifecycle: created, running, suspended, exhausted, and StopIteration.
  • send() to push values back into a running generator and throw() to raise inside it.
  • yield from for delegating to a sub-generator.
  • Generator expressions versus list comprehensions and lazy pipelining.

A generator function uses yield to produce a value and pause. When called, it returns a generator object — an iterator — without running the body yet. Each next() resumes execution from where it paused, runs until the next yield, and returns the yielded value. When the function returns (or runs off the end), the generator raises StopIteration and is exhausted. Because it produces one value at a time and holds no full list, a generator can represent infinite or very large sequences with constant memory. send() lets you push values back in; yield from delegates to another generator; generator expressions give the same laziness in a compact syntax.

Building a sequence up front means materializing it all in memory: a list of a million orders, a file read into a list of lines, the full result set of a database query. For large or streaming data, that wastes memory and adds latency before the first item is usable. The alternative without generators is a callback or a custom iterator class with __iter__ and __next__ — verbose and easy to get wrong.

A generator function replaces that with ordinary-looking code: yield produces each item, the function suspends between yields, and the caller drives it with next() or a for loop. The function’s local variables survive the suspension, so you keep readable state without a class. The caller consumes items lazily, so a million-row stream uses roughly constant memory, and the first item is available before the last is produced.

def count_up_to(n):
i = 0
while i < n:
yield i
i += 1
gen = count_up_to(3) # nothing runs yet
print(next(gen)) # 0 -- runs to first yield, pauses
print(next(gen)) # 1 -- resumes, runs to next yield
print(next(gen)) # 2
print(next(gen)) # raises StopIteration

Calling count_up_to(3) does not run the body; it returns a generator object. next() runs to the first yield and returns the yielded value; the function is suspended there. The next next() resumes just after that yield. When the function returns, next() raises StopIteration, signaling the end.

for value in count_up_to(3):
print(value)
# 0 1 2

A for loop calls next() for you and stops on StopIteration. next(gen, default) returns default instead of raising — handy when you want “the first item or a fallback.”

def running_average():
total = 0.0
count = 0
average = 0.0
while True:
value = yield average
total += value
count += 1
average = total / count
avg = running_average()
print(next(avg)) # 0.0 -- prime: advance to first yield
print(avg.send(10)) # 10.0
print(avg.send(20)) # 15.0
print(avg.send(30)) # 20.0

send(value) resumes the generator, with value becoming the result of the yield expression that paused it. The generator must be primed first — advanced to the first yield — usually with next(gen) or gen.send(None). After send resumes, the generator runs to the next yield and returns that value.

e.throw(ValueError, "bad input")

throw(type, value) raises the exception at the paused yield. If the generator catches it, execution continues; otherwise it propagates out.

def chained(a, b):
yield from a
yield from b
print(list(chained([1, 2], [3, 4]))) # [1, 2, 3, 4]

yield from iterates a sub-generator and yields each of its values, forwarding send and throw to it. It flattens nested iteration and lets a generator compose others cleanly.

squares_list = [x * x for x in range(5)] # list, all materialized
squares_gen = (x * x for x in range(5)) # generator, lazy
print(sum(squares_gen)) # 30

Parentheses make a generator expression; it yields one value at a time. It is to a list comprehension what a generator function is to one that returns a list. As the only argument to a call, the generator’s own parentheses are enough: sum(x for x in range(5)).

def naturals(start=1):
n = start
while True:
yield n
n += 1

Because nothing is stored, an infinite generator is fine — you just never iterate it to exhaustion; take what you need with islice or a bounded loop.

Result shape and edge cases:

  • A generator is single-use: once exhausted, next() keeps raising StopIteration; there is no rewind. To re-iterate, call the function again for a fresh generator.
  • Calling next() on an un-primed generator that uses send runs the body up to the first yield and returns that yield’s value (send(None) is equivalent to next()).
  • A return value inside a generator becomes the .value attribute of the StopIteration, accessible via yield from or by catching StopIteration. Returning a value from a plain generator is rare in application code.
  • Generators do not support indexing or slicing — gen[0] and gen[1:3] raise TypeError. Use itertools.islice.

OrdersPortal streams order records from a large JSON-lines file, filters paid orders, and computes a running total — without ever holding the whole file in memory.

import json
def read_orders(path):
"""Yield one order dict per line from a JSON-lines file."""
with open(path, encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
yield json.loads(line)
def paid_orders(orders):
for order in orders:
if order.get("status") == "paid":
yield order
def running_totals(orders):
total = 0.0
for order in orders:
total += float(order["amount"])
yield order["id"], total
# Compose a lazy pipeline: file -> dicts -> paid -> running total
# pipeline = running_totals(paid_orders(read_orders("orders.jsonl")))
# for order_id, total in pipeline:
# print(order_id, total)

Each stage is a generator; chaining them processes one order fully before the next is read, so the memory footprint stays flat regardless of file size. A bounded consumer:

from itertools import islice
def first_n_paid(path, n):
return list(islice(paid_orders(read_orders(path)), n))

A coroutine that accepts order amounts pushed in via send and yields the running average — the same pattern as above, generalized:

def running_average():
total = 0.0
count = 0
average = 0.0
while True:
value = yield average
total += value
count += 1
average = total / count
avg = running_average()
next(avg) # prime
print(avg.send(10)) # 10.0
print(avg.send(20)) # 15.0
print(avg.send(30)) # 20.0
  • Pipeline generators for streaming ETL: read, transform, filter, and aggregate, each a generator, so memory stays flat.
  • Generator expressions for one-pass aggregates: sum(x for x in ...), max(...) — avoids an intermediate list.
  • islice to bound a huge generator instead of slicing (generators do not support [a:b]).
  • Prime a coroutine with next(gen) before send; a small wrapper or context manager can enforce this.
  • yield from to compose generators and flatten nested iteration; it forwards send and throw, so it composes coroutines too.
  • Trying to index or slice a generator: gen[0] and gen[1:3] raise TypeError. Use islice or convert with list().
  • Reusing an exhausted generator — it is single-use; call the function again for a fresh one.
  • Calling send() on an un-primed generator — raises TypeError: can't send non-None value to a just-started generator. Prime with next(gen) or gen.send(None) first.
  • Materializing a huge generator with list(gen) and blowing memory — the laziness is the point; consume in a loop.
  • Confusing yield (produces a value, pauses) with return (ends the generator). return x sets StopIteration.value.
  • Forgetting that send resumes at the yield that paused, so the value you send becomes the result of that yield expression — not the next one.

Use generators when:

  • The sequence is large, streaming, or infinite and you want constant memory.
  • You are building a multi-stage pipeline (read, filter, transform, aggregate).
  • A custom iterator would need persistent state that fits naturally as local variables.
  • You need a coroutine that receives values over time (send).

Avoid generators when:

  • You need random access or repeated iteration — build a list instead.
  • The sequence is small and you reuse it — a list comprehension is simpler and re-iterable.
  • You need len() or indexing — generators do not support these.

Generators and yield have been in Python since 2.2 (PEP 255). yield from arrived in 3.3 (PEP 380). send and throw since 2.5. Generator expressions since 2.4. Returning a value from a generator (return x) since 3.3. Everything here runs on 3.9 through 3.14. Asynchronous generators (async def with yield, PEP 525) are a separate 3.6+ feature for async iteration.

  1. What happens when you call a generator function — does its body run immediately? No. Calling a generator function returns a generator object without executing any code. The body runs only when you call next() on the object (directly or via a for loop), and it executes up to the first yield, then suspends. Each subsequent next() resumes just after the yield that paused it.

  2. What is the difference between a generator expression and a list comprehension, and when does memory matter? [x for x in r] builds the whole list in memory before returning it; (x for x in r) returns a generator that yields one item at a time. For a million-item source consumed once, the list comprehension allocates a million-item list; the generator uses roughly constant memory. Use the list when you need indexing, len, or repeated iteration; use the generator for one-pass streaming or large sources.

  3. What does yield from do, and how does it interact with send and throw? yield from iterates a sub-iterator and yields each of its values, so the outer generator acts as a pass-through. It also forwards send() and throw() to the sub-generator and makes the sub-generator’s return value the value of the yield from expression. That makes it the right tool for composing generators and coroutines, not just flattening iteration.

  4. Senior scenario: you stream orders from a 5 GB JSON-lines file, filter and transform with chained generators, and the team later needs the transformed results twice — once to compute a total and once to write to a CSV. The second loop over the pipeline produces nothing. Why, and what are your options? Generators are single-use: the pipeline is itself a generator (each stage yields from the previous), so once the first loop drives it to exhaustion, the second loop gets StopIteration immediately and produces nothing. Options: (a) materialize once with transformed = list(pipeline) and iterate the list twice — simple, but it holds all transformed orders in memory, which defeats the streaming benefit on 5 GB; (b) re-create the pipeline for the second pass by calling the generator functions again (re-reading the file), keeping memory flat at the cost of a second I/O pass; (c) compute the total while writing the CSV in a single pass, accumulating the total as you write each row, so you only consume the stream once. Option (c) is usually best: a single pass that both writes and accumulates keeps memory flat and avoids a second read.

  • Calling a generator function returns an iterator; the body runs lazily on next().
  • Generators hold roughly constant memory and can represent infinite sequences.
  • send() pushes values back in (after priming); throw() raises inside; yield from delegates.
  • Generator expressions are the lazy, parenthesized form of list comprehensions.
  • Generators are single-use — re-iterate by recreating, or materialize a list if you need to reuse.
  • itertools — iterator building blocks that compose with generators
  • lru_cache — memoization, a complement to lazy generation
  • context managers — generator-based context managers with @contextmanager
  • List comprehensions — the eager counterpart to generator expressions
  • async/await — async generators for async iteration