Skip to content

Python Iterators and Generators: yield and Lazy Evaluation

Most of your loops run over lists, but lists have a cost: they hold every value in memory at once. Iterators and generators flip that around by producing values one at a time, only when asked. That laziness lets you stream a million order records, model infinite sequences, and chain pipelines without blowing up memory. This tutorial uses OrdersPortal order records to walk through the iterator protocol, yield, generator expressions, and a few itertools helpers.

  • The iterator protocol: __iter__ and __next__, and how iter() and next() drive it.
  • How StopIteration signals the end of a sequence.
  • How generator functions use yield to produce values lazily, one at a time.
  • How generator expressions save memory and how itertools extends iteration.
  • Python 3.9 or newer installed (run python --version to check).
  • Comfort with for loops, lists, and functions.
  • A way to run a script: save code as .py and run python filename.py.

An iterable is anything you can loop over: a list, str, dict, or a file object. An iterator is the object that actually produces the next value. iter(iterable) returns an iterator; next(iterator) returns the next value. When the iterator runs out, next raises StopIteration.

orders = ["ORD-001", "ORD-002", "ORD-003"]
it = iter(orders)
print(next(it)) # ORD-001
print(next(it)) # ORD-002
print(next(it)) # ORD-003

Verify: The script prints the three IDs in order. One more next(it) raises StopIteration.

A class becomes an iterator when it defines __iter__ (returning the iterator, usually self) and __next__ (returning the next value or raising StopIteration). This is the protocol for loops rely on.

class Countdown:
def __init__(self, start: int):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for n in Countdown(3):
print(n)

Verify: The loop prints 3, 2, 1 on separate lines, then stops.

A generator function is a concise iterator. Instead of a class, write a function that uses yield to emit each value. Calling the function returns a generator object; the function pauses at each yield and resumes there on the next call to next.

def order_ids(start: int, count: int):
for i in range(count):
yield f"ORD-{start + i:03d}"
for oid in order_ids(1, 3):
print(oid)

Verify: The loop prints ORD-001, ORD-002, ORD-003. The body runs lazily, so it does not build the whole list first.

Because a generator produces values on demand, it can represent sequences too large to hold in memory. The work happens only as you consume values.

def all_order_numbers():
n = 1
while True:
yield f"ORD-{n:06d}"
n += 1
import itertools
for oid in itertools.islice(all_order_numbers(), 3):
print(oid)

Verify: The loop prints ORD-000001, ORD-000002, ORD-000003. The generator is infinite, but islice takes only the first three.

A generator expression looks like a list comprehension but uses parentheses and computes each value lazily. It uses almost no memory when the consumer only needs to iterate once, which is the common case for sum, max, and any.

orders = [149.99, 49.99, 19.99, 299.00]
# List comprehension builds the whole list in memory
squares_list = [amt ** 2 for amt in orders]
# Generator expression computes on demand
total = sum(amt for amt in orders) # no intermediate list
big_count = sum(1 for amt in orders if amt > 100)
print(total, big_count)

Verify: The script prints 518.97 2. The generator expression passes values to sum one at a time.

yield from yields every value from a sub-iterable in one statement. It flattens nested iteration and reads more clearly than a nested loop.

def all_ids(batches):
for batch in batches:
yield from batch
batches = [["ORD-001", "ORD-002"], ["ORD-003"], ["ORD-004", "ORD-005"]]
print(list(all_ids(batches)))

Verify: The script prints ['ORD-001', 'ORD-002', 'ORD-003', 'ORD-004', 'ORD-005'].

The itertools module provides building blocks for common iteration patterns. Learn a handful and you will write less loop code.

import itertools
print(list(itertools.chain([1, 2], [3, 4]))) # [1, 2, 3, 4]
print(list(itertools.islice(range(100), 5))) # [0, 1, 2, 3, 4]
print(list(itertools.repeat("ORD", 3))) # ['ORD', 'ORD', 'ORD']

Verify: The script prints the three lists shown in the comments.

Save this as orders_stream.py and run python orders_stream.py.

"""Stream order records and summarize them lazily."""
import itertools
def read_order_records(records):
"""Yield order dicts one at a time from a list of raw records."""
for rec in records:
yield {
"id": rec["id"],
"amount": rec["amount"],
"status": rec["status"],
}
def shipped_totals(records):
"""Yield only the amounts of shipped orders."""
for rec in records:
if rec["status"] == "shipped":
yield rec["amount"]
def summarize(records) -> tuple[int, float]:
"""Return count of shipped orders and their combined total."""
totals = list(shipped_totals(records))
return len(totals), round(sum(totals), 2)
if __name__ == "__main__":
raw = [
{"id": "ORD-001", "amount": 149.99, "status": "shipped"},
{"id": "ORD-002", "amount": 49.99, "status": "pending"},
{"id": "ORD-003", "amount": 299.00, "status": "shipped"},
{"id": "ORD-004", "amount": 19.99, "status": "cancelled"},
]
records = read_order_records(raw)
# Take only the first two records as a pipeline demo
head = itertools.islice(records, 2)
count, total = summarize(head)
print(f"{count} shipped, total {total:.2f}")

Expected output:

1 shipped, total 149.99
  • Confusing an iterable with an iterator. A list is iterable but not an iterator. You can loop over a list many times, but an iterator is exhausted after one pass.
  • Reusing an exhausted iterator. After a for loop finishes, the iterator is empty. Get a fresh one with iter() or call the generator function again.
  • Calling next() without handling the end. Iterating with a for loop handles StopIteration for you. Calling next() directly means you must catch StopIteration or pass a default: next(it, None).
  • Building a huge list when a generator would do. sum([x for x in range(10**9)]) allocates billions of items. sum(x for x in range(10**9)) uses a generator and almost no memory.
  • Forgetting that generators run only when consumed. A generator function with yield does nothing until you iterate it. Assigning it to a variable just creates the object.
  • Comprehensions intro — list, dict, set comprehensions, and generator expressions in depth.
  • Loops — for and while, which are built on the iterator protocol.
  • Functions basics — generator functions are regular functions that use yield.
  • Python basics index — overview of every basics page in one place.
  • Lists — the iterable you will convert to and from most often.
  • Dictionaries — iterating keys, values, and items.