Python Generators: yield, send, and Lazy Evaluation
What you’ll learn
Section titled “What you’ll learn”- What a generator function is and how
yieldsuspends and resumes execution. - The generator lifecycle: created, running, suspended, exhausted, and
StopIteration. send()to push values back into a running generator andthrow()to raise inside it.yield fromfor delegating to a sub-generator.- Generator expressions versus list comprehensions and lazy pipelining.
Summary
Section titled “Summary”A generator function uses yield to produce a value and pause. When called, it returns a generator object — an iterator — without running the body yet. Each next() resumes execution from where it paused, runs until the next yield, and returns the yielded value. When the function returns (or runs off the end), the generator raises StopIteration and is exhausted. Because it produces one value at a time and holds no full list, a generator can represent infinite or very large sequences with constant memory. send() lets you push values back in; yield from delegates to another generator; generator expressions give the same laziness in a compact syntax.
The problem it solves
Section titled “The problem it solves”Building a sequence up front means materializing it all in memory: a list of a million orders, a file read into a list of lines, the full result set of a database query. For large or streaming data, that wastes memory and adds latency before the first item is usable. The alternative without generators is a callback or a custom iterator class with __iter__ and __next__ — verbose and easy to get wrong.
A generator function replaces that with ordinary-looking code: yield produces each item, the function suspends between yields, and the caller drives it with next() or a for loop. The function’s local variables survive the suspension, so you keep readable state without a class. The caller consumes items lazily, so a million-row stream uses roughly constant memory, and the first item is available before the last is produced.
How it works
Section titled “How it works”yield and the generator object
Section titled “yield and the generator object”def count_up_to(n): i = 0 while i < n: yield i i += 1
gen = count_up_to(3) # nothing runs yetprint(next(gen)) # 0 -- runs to first yield, pausesprint(next(gen)) # 1 -- resumes, runs to next yieldprint(next(gen)) # 2print(next(gen)) # raises StopIterationCalling count_up_to(3) does not run the body; it returns a generator object. next() runs to the first yield and returns the yielded value; the function is suspended there. The next next() resumes just after that yield. When the function returns, next() raises StopIteration, signaling the end.
Consuming with a for loop
Section titled “Consuming with a for loop”for value in count_up_to(3): print(value)# 0 1 2A for loop calls next() for you and stops on StopIteration. next(gen, default) returns default instead of raising — handy when you want “the first item or a fallback.”
send() — pushing values in
Section titled “send() — pushing values in”def running_average(): total = 0.0 count = 0 average = 0.0 while True: value = yield average total += value count += 1 average = total / count
avg = running_average()print(next(avg)) # 0.0 -- prime: advance to first yieldprint(avg.send(10)) # 10.0print(avg.send(20)) # 15.0print(avg.send(30)) # 20.0send(value) resumes the generator, with value becoming the result of the yield expression that paused it. The generator must be primed first — advanced to the first yield — usually with next(gen) or gen.send(None). After send resumes, the generator runs to the next yield and returns that value.
throw() — raising inside
Section titled “throw() — raising inside”e.throw(ValueError, "bad input")throw(type, value) raises the exception at the paused yield. If the generator catches it, execution continues; otherwise it propagates out.
yield from — delegation
Section titled “yield from — delegation”def chained(a, b): yield from a yield from b
print(list(chained([1, 2], [3, 4]))) # [1, 2, 3, 4]yield from iterates a sub-generator and yields each of its values, forwarding send and throw to it. It flattens nested iteration and lets a generator compose others cleanly.
Generator expressions
Section titled “Generator expressions”squares_list = [x * x for x in range(5)] # list, all materializedsquares_gen = (x * x for x in range(5)) # generator, lazyprint(sum(squares_gen)) # 30Parentheses make a generator expression; it yields one value at a time. It is to a list comprehension what a generator function is to one that returns a list. As the only argument to a call, the generator’s own parentheses are enough: sum(x for x in range(5)).
Infinite generators
Section titled “Infinite generators”def naturals(start=1): n = start while True: yield n n += 1Because nothing is stored, an infinite generator is fine — you just never iterate it to exhaustion; take what you need with islice or a bounded loop.
Result shape and edge cases:
- A generator is single-use: once exhausted,
next()keeps raisingStopIteration; there is no rewind. To re-iterate, call the function again for a fresh generator. - Calling
next()on an un-primed generator that usessendruns the body up to the firstyieldand returns that yield’s value (send(None)is equivalent tonext()). - A
return valueinside a generator becomes the.valueattribute of theStopIteration, accessible viayield fromor by catchingStopIteration. Returning a value from a plain generator is rare in application code. - Generators do not support indexing or slicing —
gen[0]andgen[1:3]raiseTypeError. Useitertools.islice.
Realistic example
Section titled “Realistic example”OrdersPortal streams order records from a large JSON-lines file, filters paid orders, and computes a running total — without ever holding the whole file in memory.
import json
def read_orders(path): """Yield one order dict per line from a JSON-lines file.""" with open(path, encoding="utf-8") as f: for line in f: line = line.strip() if line: yield json.loads(line)
def paid_orders(orders): for order in orders: if order.get("status") == "paid": yield order
def running_totals(orders): total = 0.0 for order in orders: total += float(order["amount"]) yield order["id"], total
# Compose a lazy pipeline: file -> dicts -> paid -> running total# pipeline = running_totals(paid_orders(read_orders("orders.jsonl")))# for order_id, total in pipeline:# print(order_id, total)Each stage is a generator; chaining them processes one order fully before the next is read, so the memory footprint stays flat regardless of file size. A bounded consumer:
from itertools import islice
def first_n_paid(path, n): return list(islice(paid_orders(read_orders(path)), n))A coroutine that accepts order amounts pushed in via send and yields the running average — the same pattern as above, generalized:
def running_average(): total = 0.0 count = 0 average = 0.0 while True: value = yield average total += value count += 1 average = total / count
avg = running_average()next(avg) # primeprint(avg.send(10)) # 10.0print(avg.send(20)) # 15.0print(avg.send(30)) # 20.0Production patterns
Section titled “Production patterns”- Pipeline generators for streaming ETL: read, transform, filter, and aggregate, each a generator, so memory stays flat.
- Generator expressions for one-pass aggregates:
sum(x for x in ...),max(...)— avoids an intermediate list. isliceto bound a huge generator instead of slicing (generators do not support[a:b]).- Prime a coroutine with
next(gen)beforesend; a small wrapper or context manager can enforce this. yield fromto compose generators and flatten nested iteration; it forwardssendandthrow, so it composes coroutines too.
Common mistakes
Section titled “Common mistakes”- Trying to index or slice a generator:
gen[0]andgen[1:3]raiseTypeError. Useisliceor convert withlist(). - Reusing an exhausted generator — it is single-use; call the function again for a fresh one.
- Calling
send()on an un-primed generator — raisesTypeError: can't send non-None value to a just-started generator. Prime withnext(gen)orgen.send(None)first. - Materializing a huge generator with
list(gen)and blowing memory — the laziness is the point; consume in a loop. - Confusing
yield(produces a value, pauses) withreturn(ends the generator).return xsetsStopIteration.value. - Forgetting that
sendresumes at theyieldthat paused, so the value you send becomes the result of thatyieldexpression — not the next one.
When to use it and when not to
Section titled “When to use it and when not to”Use generators when:
- The sequence is large, streaming, or infinite and you want constant memory.
- You are building a multi-stage pipeline (read, filter, transform, aggregate).
- A custom iterator would need persistent state that fits naturally as local variables.
- You need a coroutine that receives values over time (
send).
Avoid generators when:
- You need random access or repeated iteration — build a list instead.
- The sequence is small and you reuse it — a list comprehension is simpler and re-iterable.
- You need
len()or indexing — generators do not support these.
Compatibility
Section titled “Compatibility”Generators and yield have been in Python since 2.2 (PEP 255). yield from arrived in 3.3 (PEP 380). send and throw since 2.5. Generator expressions since 2.4. Returning a value from a generator (return x) since 3.3. Everything here runs on 3.9 through 3.14. Asynchronous generators (async def with yield, PEP 525) are a separate 3.6+ feature for async iteration.
Interview questions
Section titled “Interview questions”-
What happens when you call a generator function — does its body run immediately? No. Calling a generator function returns a generator object without executing any code. The body runs only when you call
next()on the object (directly or via aforloop), and it executes up to the firstyield, then suspends. Each subsequentnext()resumes just after theyieldthat paused it. -
What is the difference between a generator expression and a list comprehension, and when does memory matter?
[x for x in r]builds the whole list in memory before returning it;(x for x in r)returns a generator that yields one item at a time. For a million-item source consumed once, the list comprehension allocates a million-item list; the generator uses roughly constant memory. Use the list when you need indexing,len, or repeated iteration; use the generator for one-pass streaming or large sources. -
What does
yield fromdo, and how does it interact withsendandthrow?yield fromiterates a sub-iterator and yields each of its values, so the outer generator acts as a pass-through. It also forwardssend()andthrow()to the sub-generator and makes the sub-generator’s return value the value of theyield fromexpression. That makes it the right tool for composing generators and coroutines, not just flattening iteration. -
Senior scenario: you stream orders from a 5 GB JSON-lines file, filter and transform with chained generators, and the team later needs the transformed results twice — once to compute a total and once to write to a CSV. The second loop over the pipeline produces nothing. Why, and what are your options? Generators are single-use: the pipeline is itself a generator (each stage yields from the previous), so once the first loop drives it to exhaustion, the second loop gets
StopIterationimmediately and produces nothing. Options: (a) materialize once withtransformed = list(pipeline)and iterate the list twice — simple, but it holds all transformed orders in memory, which defeats the streaming benefit on 5 GB; (b) re-create the pipeline for the second pass by calling the generator functions again (re-reading the file), keeping memory flat at the cost of a second I/O pass; (c) compute the total while writing the CSV in a single pass, accumulating the total as you write each row, so you only consume the stream once. Option (c) is usually best: a single pass that both writes and accumulates keeps memory flat and avoids a second read.
Key takeaways
Section titled “Key takeaways”- Calling a generator function returns an iterator; the body runs lazily on
next(). - Generators hold roughly constant memory and can represent infinite sequences.
send()pushes values back in (after priming);throw()raises inside;yield fromdelegates.- Generator expressions are the lazy, parenthesized form of list comprehensions.
- Generators are single-use — re-iterate by recreating, or materialize a list if you need to reuse.
Related pages
Section titled “Related pages”- itertools — iterator building blocks that compose with generators
- lru_cache — memoization, a complement to lazy generation
- context managers — generator-based context managers with
@contextmanager - List comprehensions — the eager counterpart to generator expressions
- async/await — async generators for async iteration