Python Iterators and Generators: yield and Lazy Evaluation
Most of your loops run over lists, but lists have a cost: they hold every value in memory at once. Iterators and generators flip that around by producing values one at a time, only when asked. That laziness lets you stream a million order records, model infinite sequences, and chain pipelines without blowing up memory. This tutorial uses OrdersPortal order records to walk through the iterator protocol, yield, generator expressions, and a few itertools helpers.
What you’ll learn
Section titled “What you’ll learn”- The iterator protocol:
__iter__and__next__, and howiter()andnext()drive it. - How
StopIterationsignals the end of a sequence. - How generator functions use
yieldto produce values lazily, one at a time. - How generator expressions save memory and how
itertoolsextends iteration.
Prerequisites
Section titled “Prerequisites”- Python 3.9 or newer installed (run
python --versionto check). - Comfort with
forloops, lists, and functions. - A way to run a script: save code as
.pyand runpython filename.py.
1. Tell iterables from iterators
Section titled “1. Tell iterables from iterators”An iterable is anything you can loop over: a list, str, dict, or a file object. An iterator is the object that actually produces the next value. iter(iterable) returns an iterator; next(iterator) returns the next value. When the iterator runs out, next raises StopIteration.
orders = ["ORD-001", "ORD-002", "ORD-003"]it = iter(orders)
print(next(it)) # ORD-001print(next(it)) # ORD-002print(next(it)) # ORD-003Verify: The script prints the three IDs in order. One more next(it) raises StopIteration.
2. Implement the iterator protocol
Section titled “2. Implement the iterator protocol”A class becomes an iterator when it defines __iter__ (returning the iterator, usually self) and __next__ (returning the next value or raising StopIteration). This is the protocol for loops rely on.
class Countdown: def __init__(self, start: int): self.current = start
def __iter__(self): return self
def __next__(self): if self.current <= 0: raise StopIteration value = self.current self.current -= 1 return value
for n in Countdown(3): print(n)Verify: The loop prints 3, 2, 1 on separate lines, then stops.
3. Build a generator with yield
Section titled “3. Build a generator with yield”A generator function is a concise iterator. Instead of a class, write a function that uses yield to emit each value. Calling the function returns a generator object; the function pauses at each yield and resumes there on the next call to next.
def order_ids(start: int, count: int): for i in range(count): yield f"ORD-{start + i:03d}"
for oid in order_ids(1, 3): print(oid)Verify: The loop prints ORD-001, ORD-002, ORD-003. The body runs lazily, so it does not build the whole list first.
4. Defer work with lazy evaluation
Section titled “4. Defer work with lazy evaluation”Because a generator produces values on demand, it can represent sequences too large to hold in memory. The work happens only as you consume values.
def all_order_numbers(): n = 1 while True: yield f"ORD-{n:06d}" n += 1
import itertoolsfor oid in itertools.islice(all_order_numbers(), 3): print(oid)Verify: The loop prints ORD-000001, ORD-000002, ORD-000003. The generator is infinite, but islice takes only the first three.
5. Use generator expressions
Section titled “5. Use generator expressions”A generator expression looks like a list comprehension but uses parentheses and computes each value lazily. It uses almost no memory when the consumer only needs to iterate once, which is the common case for sum, max, and any.
orders = [149.99, 49.99, 19.99, 299.00]
# List comprehension builds the whole list in memorysquares_list = [amt ** 2 for amt in orders]
# Generator expression computes on demandtotal = sum(amt for amt in orders) # no intermediate listbig_count = sum(1 for amt in orders if amt > 100)print(total, big_count)Verify: The script prints 518.97 2. The generator expression passes values to sum one at a time.
6. Delegate with yield from
Section titled “6. Delegate with yield from”yield from yields every value from a sub-iterable in one statement. It flattens nested iteration and reads more clearly than a nested loop.
def all_ids(batches): for batch in batches: yield from batch
batches = [["ORD-001", "ORD-002"], ["ORD-003"], ["ORD-004", "ORD-005"]]print(list(all_ids(batches)))Verify: The script prints ['ORD-001', 'ORD-002', 'ORD-003', 'ORD-004', 'ORD-005'].
7. Reach for itertools
Section titled “7. Reach for itertools”The itertools module provides building blocks for common iteration patterns. Learn a handful and you will write less loop code.
import itertools
print(list(itertools.chain([1, 2], [3, 4]))) # [1, 2, 3, 4]print(list(itertools.islice(range(100), 5))) # [0, 1, 2, 3, 4]print(list(itertools.repeat("ORD", 3))) # ['ORD', 'ORD', 'ORD']Verify: The script prints the three lists shown in the comments.
Complete working example
Section titled “Complete working example”Save this as orders_stream.py and run python orders_stream.py.
"""Stream order records and summarize them lazily."""import itertools
def read_order_records(records): """Yield order dicts one at a time from a list of raw records.""" for rec in records: yield { "id": rec["id"], "amount": rec["amount"], "status": rec["status"], }
def shipped_totals(records): """Yield only the amounts of shipped orders.""" for rec in records: if rec["status"] == "shipped": yield rec["amount"]
def summarize(records) -> tuple[int, float]: """Return count of shipped orders and their combined total.""" totals = list(shipped_totals(records)) return len(totals), round(sum(totals), 2)
if __name__ == "__main__": raw = [ {"id": "ORD-001", "amount": 149.99, "status": "shipped"}, {"id": "ORD-002", "amount": 49.99, "status": "pending"}, {"id": "ORD-003", "amount": 299.00, "status": "shipped"}, {"id": "ORD-004", "amount": 19.99, "status": "cancelled"}, ]
records = read_order_records(raw) # Take only the first two records as a pipeline demo head = itertools.islice(records, 2) count, total = summarize(head) print(f"{count} shipped, total {total:.2f}")Expected output:
1 shipped, total 149.99Common mistakes
Section titled “Common mistakes”- Confusing an iterable with an iterator. A
listis iterable but not an iterator. You can loop over alistmany times, but an iterator is exhausted after one pass. - Reusing an exhausted iterator. After a
forloop finishes, the iterator is empty. Get a fresh one withiter()or call the generator function again. - Calling
next()without handling the end. Iterating with aforloop handlesStopIterationfor you. Callingnext()directly means you must catchStopIterationor pass a default:next(it, None). - Building a huge list when a generator would do.
sum([x for x in range(10**9)])allocates billions of items.sum(x for x in range(10**9))uses a generator and almost no memory. - Forgetting that generators run only when consumed. A generator function with
yielddoes nothing until you iterate it. Assigning it to a variable just creates the object.
Next steps
Section titled “Next steps”- Comprehensions intro — list, dict, set comprehensions, and generator expressions in depth.
- Loops —
forandwhile, which are built on the iterator protocol. - Functions basics — generator functions are regular functions that use
yield.
Related pages
Section titled “Related pages”- Python basics index — overview of every basics page in one place.
- Lists — the iterable you will convert to and from most often.
- Dictionaries — iterating keys, values, and items.
Official references
Section titled “Official references”- docs.python.org: Iterators — the iterator protocol tutorial.
- docs.python.org: Generators —
yieldand generator functions. - docs.python.org: Generator expressions — the glossary definition and examples.
- docs.python.org: The
itertoolsmodule — iterator building blocks.