Skip to content

Python itertools: Iterator Algebra

  • The lazy, memory-efficient model behind itertools and why it matters.
  • Chaining and slicing iterators: chain, islice, chain.from_iterable.
  • Grouping and accumulating: groupby, accumulate, starmap.
  • Filtering with predicates: takewhile, dropwhile.
  • Infinite iterators and combinatorics: count/cycle/repeat, product/permutations/combinations, and tee.

itertools is a standard-library module of fast, memory-efficient iterator building blocks. Every function returns a lazy iterator that yields one item at a time, so you can chain, slice, group, and combine very large (or infinite) sequences without materializing them. chain concatenates iterables; islice slices an iterator by index or range; groupby groups consecutive equal keys; accumulate runs a running reduction; takewhile/dropwhile filter by a predicate; count/cycle/repeat produce infinite streams; product/permutations/combinations enumerate the combinatoric possibilities; tee duplicates an iterator. Composed together, they replace many explicit loops with one-liners that stay lazy end to end.

Hand-written loops over sequences tend to materialize intermediate results: concatenate lists with +, slice with [a:b], group with a loop and a dict, accumulate with a running variable. Each step builds a new list, so a pipeline of three steps holds three copies in memory and waits for each stage to finish before the next starts. For large or streaming data that is slow and memory-hungry.

itertools functions return iterators, not lists, and they compose: chain(islice(...), takewhile(...)) produces a single iterator that pulls one item at a time through every stage. Nothing is materialized until you consume it, so memory stays flat and the first item is produced before the last input is read. It also expresses intent: islice(stream, 10) says “first 10”; takewhile(pred, stream) says “until the predicate fails.”

from itertools import chain
list(chain([1, 2], [3, 4], [5])) # [1, 2, 3, 4, 5]
matrix = [[1, 2], [3, 4], [5, 6]]
list(chain.from_iterable(matrix)) # [1, 2, 3, 4, 5, 6] -- flatten one level

chain takes iterables as separate arguments; chain.from_iterable takes a single iterable of iterables.

from itertools import islice
list(islice(range(1000), 5)) # [0, 1, 2, 3, 4]
list(islice(range(1000), 2, 6)) # [2, 3, 4, 5]
list(islice(range(1000), 0, 10, 2)) # [0, 2, 4, 6, 8]

islice works on any iterator (generators do not support [a:b]); it consumes items lazily and stops once the slice is satisfied.

from itertools import groupby
rows = [("paid", "ORD-1"), ("paid", "ORD-2"), ("shipped", "ORD-3")]
for status, group in groupby(rows, key=lambda r: r[0]):
print(status, [r[1] for r in group])
# paid ['ORD-1', 'ORD-2']
# shipped ['ORD-3']

groupby groups consecutive runs of equal keys. If the input is not sorted by the key, equal keys in different places form separate groups — sort first if you want all equal keys together.

from itertools import accumulate
import operator
list(accumulate([1, 2, 3, 4])) # [1, 3, 6, 10] -- running sum
list(accumulate([1, 2, 3, 4], operator.mul)) # [1, 2, 6, 24] -- running product

The default is sum; pass any binary function. (Python 3.8+ supports an initial value.)

from itertools import starmap
pairs = [(2, 3), (4, 5)]
list(starmap(lambda a, b: a * b, pairs)) # [6, 20]

starmap applies a function to tuples, unpacking each one — like map(*tup) for every item.

takewhile / dropwhile — predicate filtering

Section titled “takewhile / dropwhile — predicate filtering”
from itertools import takewhile, dropwhile
list(takewhile(lambda x: x < 3, [1, 2, 3, 4, 1])) # [1, 2] -- stops at first failure
list(dropwhile(lambda x: x < 3, [1, 2, 3, 4, 1])) # [3, 4, 1] -- skips the prefix, keeps the rest

takewhile yields until the predicate is false, then stops (it does not resume). dropwhile skips the leading run where the predicate is true, then yields everything after.

count / cycle / repeat — infinite iterators

Section titled “count / cycle / repeat — infinite iterators”
from itertools import count, cycle, repeat
# count() -> 0, 1, 2, ... (optional start, step)
# cycle("AB") -> A, B, A, B, ...
# repeat("x", 3) -> x, x, x (or infinite with no count)

Use with islice to bound them: list(islice(count(10), 5)) is [10, 11, 12, 13, 14].

product / permutations / combinations — combinatorics

Section titled “product / permutations / combinations — combinatorics”
from itertools import product, permutations, combinations
list(product("AB", repeat=2)) # [('A','A'),('A','B'),('B','A'),('B','B')]
list(permutations("AB", 2)) # [('A','B'),('B','A')]
list(combinations("ABC", 2)) # [('A','B'),('A','C'),('B','C')]

product is the Cartesian product (with repeat for sampling with replacement); permutations is order-sensitive without replacement; combinations is order-insensitive without replacement.

from itertools import tee
a, b = tee(range(3))
list(a) # [0, 1, 2]
list(b) # [0, 1, 2]

tee splits one iterator into n independent iterators. It caches items as needed, so it is memory-efficient only when the copies stay close together; if one copy runs far ahead, the cache grows.

Result shape and edge cases:

  • All itertools functions return iterators, single-use and exhausted after one pass; wrap with list() to materialize.
  • groupby groups consecutive runs; sort by the key first for a full grouping.
  • takewhile stops permanently at the first false; it is not “filter while true everywhere.”
  • count has no stop; always pair it with islice or a bounded consumer.
  • tee caches; do not advance one copy far beyond the others.

OrdersPortal streams orders, groups consecutive paid orders by customer, accumulates a running daily total, and takes the high-value orders until the first one under a threshold — all lazily.

from itertools import groupby, accumulate, islice, takewhile
# pretend this yields orders from a file or DB cursor
def read_orders():
yield {"id": "ORD-1", "customer": "Acme", "status": "paid", "amount": 100.0}
yield {"id": "ORD-2", "customer": "Acme", "status": "paid", "amount": 50.0}
yield {"id": "ORD-3", "customer": "Globex", "status": "paid", "amount": 300.0}
yield {"id": "ORD-4", "customer": "Globex", "status": "paid", "amount": 75.0}
# 1) group consecutive paid orders by customer (input is sorted by customer)
paid = (o for o in read_orders() if o["status"] == "paid")
for customer, group in groupby(paid, key=lambda o: o["customer"]):
print(customer, [o["id"] for o in group])
# Acme ['ORD-1', 'ORD-2']
# Globex ['ORD-3', 'ORD-4']
# 2) running total of order amounts, take the first 3
amounts = (o["amount"] for o in read_orders())
running = accumulate(amounts)
print(list(islice(running, 3))) # [100.0, 150.0, 450.0]
# 3) high-value orders until the first one under $50
stream = [
{"id": "ORD-1", "amount": 100.0},
{"id": "ORD-2", "amount": 80.0},
{"id": "ORD-3", "amount": 40.0}, # first under 50 -- takewhile stops here
{"id": "ORD-4", "amount": 300.0}, # and does NOT resume for this one
]
high = takewhile(lambda o: o["amount"] >= 50.0, stream)
print([o["id"] for o in high]) # ['ORD-1', 'ORD-2']

A combinatoric example: every customer/carrier pairing for a shipping matrix.

from itertools import product
customers = ["Acme", "Globex"]
carriers = ["UPS", "FedEx"]
for customer, carrier in product(customers, carriers):
print(f"{customer} via {carrier}")
  • Chain stages as generators so the whole pipeline is lazy: read, filter, transform, aggregate, each an iterator.
  • Use islice to bound infinite or huge iterators instead of slicing (generators do not support [a:b]).
  • Sort before groupby if you need all equal keys in one group; if only consecutive runs matter, skip the sort.
  • Use the itertools recipes from the docs (flatten, pairwise, roundrobin) rather than reinventing them.
  • Prefer count() over a manual integer counter for zip-with-index or tie-breaking; it is clearer and never overflows in practice.
  • Assuming groupby groups all equal keys — it groups consecutive runs; sort first for a full grouping.
  • Treating takewhile like filter — it stops at the first false and does not resume, so later matching items are dropped.
  • Forgetting that itertools iterators are single-use — they are exhausted after one pass.
  • Using count() without a bound — infinite; pair with islice or a break.
  • Letting one tee copy run far ahead of another — the internal cache grows with the distance.
  • Calling list() early and materializing everything, defeating the laziness that is the whole point.

Use itertools when:

  • You are building a lazy pipeline over large or streaming data.
  • You need to chain, slice, group, accumulate, or combine iterators without intermediate lists.
  • You need combinatoric enumeration (product, permutations, combinations).

Avoid itertools when:

  • A list comprehension or generator expression says it more clearly for a single step.
  • The data is small and a plain list or loop is simpler to read.
  • You need random access or repeated iteration — materialize a list.

itertools is in the standard library and available on Python 3.9 through 3.14 (and far older). accumulate, chain, groupby, islice, starmap, takewhile, dropwhile, count, cycle, repeat, product, permutations, combinations, and tee are all supported. accumulate’s initial parameter (3.8+) and the strict zip-termination behavior are the only recent additions; everything here runs on 3.9+.

  1. What does itertools.chain do, and how is chain.from_iterable different? chain(*iterables) lazily concatenates several iterables passed as separate arguments, yielding each item in order without building a combined list. chain.from_iterable(iterable_of_iterables) takes a single iterable that yields the iterables to chain — the right choice when the inputs are already in a container, like flattening a list of lists.

  2. How does groupby differ from grouping in SQL or pandas? groupby groups consecutive runs of equal keys as it streams; it does not collect all items with the same key from anywhere in the input. To get a SQL-style full grouping, sort the input by the key first so equal keys are adjacent, then groupby collects them into one group.

  3. What does takewhile do, and what common misunderstanding do people have about it? takewhile(pred, iterable) yields items while the predicate is true and stops the moment it is false — permanently, even if later items would match. The common mistake is treating it like filter, which keeps every matching item; takewhile keeps only the leading run.

  4. Senior scenario: a pipeline does filtered = [x for x in data if pred(x)], then sliced = filtered[:100], then grouped = group(sliced). With a 50-million-row source this OOMs. How would you refactor with itertools, and what stays lazy? The list comprehension materializes the whole filtered result before slicing, so 50 million rows become a 50-million-element list in memory. Refactor each step to an iterator: filtered = (x for x in data if pred(x)) (a generator expression), sliced = islice(filtered, 100) (bounded and lazy), and grouped = groupby(sorted(sliced, key=...)) if grouping is needed. Now filtered never holds more than one item at a time, islice stops after 100, and the memory footprint is roughly constant regardless of the source size. Only the final, bounded set of 100 items is materialized. The whole chain stays lazy until explicit consumption, and the source can even be an infinite generator.

  • itertools functions return lazy, single-use iterators that compose into memory-flat pipelines.
  • chain concatenates; islice slices; groupby groups consecutive runs (sort first for a full grouping); accumulate reduces.
  • takewhile stops at the first false; dropwhile skips the leading true run.
  • count/cycle/repeat are infinite — always bound them, usually with islice.
  • product/permutations/combinations enumerate combinatorics; tee duplicates an iterator.
  • generators — the yield-based foundation itertools builds on
  • enumerate — index-and-value pairs, a common itertools companion
  • zip — parallel iteration; combine with count or itertools
  • heapq — priority structures; count() tie-breakers come from itertools
  • lru_cache — memoization for repeated expensive calls in pipelines