Python itertools: Iterator Algebra
What you’ll learn
Section titled “What you’ll learn”- The lazy, memory-efficient model behind
itertoolsand why it matters. - Chaining and slicing iterators:
chain,islice,chain.from_iterable. - Grouping and accumulating:
groupby,accumulate,starmap. - Filtering with predicates:
takewhile,dropwhile. - Infinite iterators and combinatorics:
count/cycle/repeat,product/permutations/combinations, andtee.
Summary
Section titled “Summary”itertools is a standard-library module of fast, memory-efficient iterator building blocks. Every function returns a lazy iterator that yields one item at a time, so you can chain, slice, group, and combine very large (or infinite) sequences without materializing them. chain concatenates iterables; islice slices an iterator by index or range; groupby groups consecutive equal keys; accumulate runs a running reduction; takewhile/dropwhile filter by a predicate; count/cycle/repeat produce infinite streams; product/permutations/combinations enumerate the combinatoric possibilities; tee duplicates an iterator. Composed together, they replace many explicit loops with one-liners that stay lazy end to end.
The problem it solves
Section titled “The problem it solves”Hand-written loops over sequences tend to materialize intermediate results: concatenate lists with +, slice with [a:b], group with a loop and a dict, accumulate with a running variable. Each step builds a new list, so a pipeline of three steps holds three copies in memory and waits for each stage to finish before the next starts. For large or streaming data that is slow and memory-hungry.
itertools functions return iterators, not lists, and they compose: chain(islice(...), takewhile(...)) produces a single iterator that pulls one item at a time through every stage. Nothing is materialized until you consume it, so memory stays flat and the first item is produced before the last input is read. It also expresses intent: islice(stream, 10) says “first 10”; takewhile(pred, stream) says “until the predicate fails.”
How it works
Section titled “How it works”chain and chain.from_iterable
Section titled “chain and chain.from_iterable”from itertools import chain
list(chain([1, 2], [3, 4], [5])) # [1, 2, 3, 4, 5]matrix = [[1, 2], [3, 4], [5, 6]]list(chain.from_iterable(matrix)) # [1, 2, 3, 4, 5, 6] -- flatten one levelchain takes iterables as separate arguments; chain.from_iterable takes a single iterable of iterables.
islice — slice an iterator
Section titled “islice — slice an iterator”from itertools import islice
list(islice(range(1000), 5)) # [0, 1, 2, 3, 4]list(islice(range(1000), 2, 6)) # [2, 3, 4, 5]list(islice(range(1000), 0, 10, 2)) # [0, 2, 4, 6, 8]islice works on any iterator (generators do not support [a:b]); it consumes items lazily and stops once the slice is satisfied.
groupby — group consecutive equal keys
Section titled “groupby — group consecutive equal keys”from itertools import groupby
rows = [("paid", "ORD-1"), ("paid", "ORD-2"), ("shipped", "ORD-3")]for status, group in groupby(rows, key=lambda r: r[0]): print(status, [r[1] for r in group])# paid ['ORD-1', 'ORD-2']# shipped ['ORD-3']groupby groups consecutive runs of equal keys. If the input is not sorted by the key, equal keys in different places form separate groups — sort first if you want all equal keys together.
accumulate — running reduction
Section titled “accumulate — running reduction”from itertools import accumulateimport operator
list(accumulate([1, 2, 3, 4])) # [1, 3, 6, 10] -- running sumlist(accumulate([1, 2, 3, 4], operator.mul)) # [1, 2, 6, 24] -- running productThe default is sum; pass any binary function. (Python 3.8+ supports an initial value.)
starmap — map with unpacked arguments
Section titled “starmap — map with unpacked arguments”from itertools import starmap
pairs = [(2, 3), (4, 5)]list(starmap(lambda a, b: a * b, pairs)) # [6, 20]starmap applies a function to tuples, unpacking each one — like map(*tup) for every item.
takewhile / dropwhile — predicate filtering
Section titled “takewhile / dropwhile — predicate filtering”from itertools import takewhile, dropwhile
list(takewhile(lambda x: x < 3, [1, 2, 3, 4, 1])) # [1, 2] -- stops at first failurelist(dropwhile(lambda x: x < 3, [1, 2, 3, 4, 1])) # [3, 4, 1] -- skips the prefix, keeps the resttakewhile yields until the predicate is false, then stops (it does not resume). dropwhile skips the leading run where the predicate is true, then yields everything after.
count / cycle / repeat — infinite iterators
Section titled “count / cycle / repeat — infinite iterators”from itertools import count, cycle, repeat
# count() -> 0, 1, 2, ... (optional start, step)# cycle("AB") -> A, B, A, B, ...# repeat("x", 3) -> x, x, x (or infinite with no count)Use with islice to bound them: list(islice(count(10), 5)) is [10, 11, 12, 13, 14].
product / permutations / combinations — combinatorics
Section titled “product / permutations / combinations — combinatorics”from itertools import product, permutations, combinations
list(product("AB", repeat=2)) # [('A','A'),('A','B'),('B','A'),('B','B')]list(permutations("AB", 2)) # [('A','B'),('B','A')]list(combinations("ABC", 2)) # [('A','B'),('A','C'),('B','C')]product is the Cartesian product (with repeat for sampling with replacement); permutations is order-sensitive without replacement; combinations is order-insensitive without replacement.
tee — duplicate an iterator
Section titled “tee — duplicate an iterator”from itertools import tee
a, b = tee(range(3))list(a) # [0, 1, 2]list(b) # [0, 1, 2]tee splits one iterator into n independent iterators. It caches items as needed, so it is memory-efficient only when the copies stay close together; if one copy runs far ahead, the cache grows.
Result shape and edge cases:
- All
itertoolsfunctions return iterators, single-use and exhausted after one pass; wrap withlist()to materialize. groupbygroups consecutive runs; sort by the key first for a full grouping.takewhilestops permanently at the first false; it is not “filter while true everywhere.”counthas no stop; always pair it withisliceor a bounded consumer.teecaches; do not advance one copy far beyond the others.
Realistic example
Section titled “Realistic example”OrdersPortal streams orders, groups consecutive paid orders by customer, accumulates a running daily total, and takes the high-value orders until the first one under a threshold — all lazily.
from itertools import groupby, accumulate, islice, takewhile
# pretend this yields orders from a file or DB cursordef read_orders(): yield {"id": "ORD-1", "customer": "Acme", "status": "paid", "amount": 100.0} yield {"id": "ORD-2", "customer": "Acme", "status": "paid", "amount": 50.0} yield {"id": "ORD-3", "customer": "Globex", "status": "paid", "amount": 300.0} yield {"id": "ORD-4", "customer": "Globex", "status": "paid", "amount": 75.0}
# 1) group consecutive paid orders by customer (input is sorted by customer)paid = (o for o in read_orders() if o["status"] == "paid")for customer, group in groupby(paid, key=lambda o: o["customer"]): print(customer, [o["id"] for o in group])# Acme ['ORD-1', 'ORD-2']# Globex ['ORD-3', 'ORD-4']
# 2) running total of order amounts, take the first 3amounts = (o["amount"] for o in read_orders())running = accumulate(amounts)print(list(islice(running, 3))) # [100.0, 150.0, 450.0]
# 3) high-value orders until the first one under $50stream = [ {"id": "ORD-1", "amount": 100.0}, {"id": "ORD-2", "amount": 80.0}, {"id": "ORD-3", "amount": 40.0}, # first under 50 -- takewhile stops here {"id": "ORD-4", "amount": 300.0}, # and does NOT resume for this one]high = takewhile(lambda o: o["amount"] >= 50.0, stream)print([o["id"] for o in high]) # ['ORD-1', 'ORD-2']A combinatoric example: every customer/carrier pairing for a shipping matrix.
from itertools import product
customers = ["Acme", "Globex"]carriers = ["UPS", "FedEx"]for customer, carrier in product(customers, carriers): print(f"{customer} via {carrier}")Production patterns
Section titled “Production patterns”- Chain stages as generators so the whole pipeline is lazy: read, filter, transform, aggregate, each an iterator.
- Use
isliceto bound infinite or huge iterators instead of slicing (generators do not support[a:b]). - Sort before
groupbyif you need all equal keys in one group; if only consecutive runs matter, skip the sort. - Use the
itertoolsrecipes from the docs (flatten,pairwise,roundrobin) rather than reinventing them. - Prefer
count()over a manual integer counter forzip-with-index or tie-breaking; it is clearer and never overflows in practice.
Common mistakes
Section titled “Common mistakes”- Assuming
groupbygroups all equal keys — it groups consecutive runs; sort first for a full grouping. - Treating
takewhilelikefilter— it stops at the first false and does not resume, so later matching items are dropped. - Forgetting that
itertoolsiterators are single-use — they are exhausted after one pass. - Using
count()without a bound — infinite; pair withisliceor abreak. - Letting one
teecopy run far ahead of another — the internal cache grows with the distance. - Calling
list()early and materializing everything, defeating the laziness that is the whole point.
When to use it and when not to
Section titled “When to use it and when not to”Use itertools when:
- You are building a lazy pipeline over large or streaming data.
- You need to chain, slice, group, accumulate, or combine iterators without intermediate lists.
- You need combinatoric enumeration (
product,permutations,combinations).
Avoid itertools when:
- A list comprehension or generator expression says it more clearly for a single step.
- The data is small and a plain list or loop is simpler to read.
- You need random access or repeated iteration — materialize a list.
Compatibility
Section titled “Compatibility”itertools is in the standard library and available on Python 3.9 through 3.14 (and far older). accumulate, chain, groupby, islice, starmap, takewhile, dropwhile, count, cycle, repeat, product, permutations, combinations, and tee are all supported. accumulate’s initial parameter (3.8+) and the strict zip-termination behavior are the only recent additions; everything here runs on 3.9+.
Interview questions
Section titled “Interview questions”-
What does
itertools.chaindo, and how ischain.from_iterabledifferent?chain(*iterables)lazily concatenates several iterables passed as separate arguments, yielding each item in order without building a combined list.chain.from_iterable(iterable_of_iterables)takes a single iterable that yields the iterables to chain — the right choice when the inputs are already in a container, like flattening a list of lists. -
How does
groupbydiffer from grouping in SQL or pandas?groupbygroups consecutive runs of equal keys as it streams; it does not collect all items with the same key from anywhere in the input. To get a SQL-style full grouping, sort the input by the key first so equal keys are adjacent, thengroupbycollects them into one group. -
What does
takewhiledo, and what common misunderstanding do people have about it?takewhile(pred, iterable)yields items while the predicate is true and stops the moment it is false — permanently, even if later items would match. The common mistake is treating it likefilter, which keeps every matching item;takewhilekeeps only the leading run. -
Senior scenario: a pipeline does
filtered = [x for x in data if pred(x)], thensliced = filtered[:100], thengrouped = group(sliced). With a 50-million-row source this OOMs. How would you refactor withitertools, and what stays lazy? The list comprehension materializes the whole filtered result before slicing, so 50 million rows become a 50-million-element list in memory. Refactor each step to an iterator:filtered = (x for x in data if pred(x))(a generator expression),sliced = islice(filtered, 100)(bounded and lazy), andgrouped = groupby(sorted(sliced, key=...))if grouping is needed. Nowfilterednever holds more than one item at a time,islicestops after 100, and the memory footprint is roughly constant regardless of the source size. Only the final, bounded set of 100 items is materialized. The whole chain stays lazy until explicit consumption, and the source can even be an infinite generator.
Key takeaways
Section titled “Key takeaways”itertoolsfunctions return lazy, single-use iterators that compose into memory-flat pipelines.chainconcatenates;isliceslices;groupbygroups consecutive runs (sort first for a full grouping);accumulatereduces.takewhilestops at the first false;dropwhileskips the leading true run.count/cycle/repeatare infinite — always bound them, usually withislice.product/permutations/combinationsenumerate combinatorics;teeduplicates an iterator.
Related pages
Section titled “Related pages”- generators — the
yield-based foundationitertoolsbuilds on - enumerate — index-and-value pairs, a common
itertoolscompanion - zip — parallel iteration; combine with
countoritertools - heapq — priority structures;
count()tie-breakers come fromitertools - lru_cache — memoization for repeated expensive calls in pipelines
Official references
Section titled “Official references”- itertools — Python docs
- itertools recipes —
flatten,pairwise,roundrobin, and more - More IterTools (third-party) — extended recipes beyond the standard library