Iterators and Generators

Introduction

What if you need to process a file with a billion lines, or an endless stream of sensor readings? You cannot build a list of everything first, since it will not fit in memory. Generators produce values one at a time, on demand. They are the Python way to work with big or infinite sequences.

How for loops really work

A for loop calls iter() on the collection to get an iterator, then calls next() on it again and again until it runs out.

An iterable is anything you can loop over (lists, strings, files, dictionaries). An iterator is the object that remembers where it is. Once it is used up, it is empty.

Generator functions with yield

Normally a function runs, returns a value, and finishes. A function that contains yield becomes a generator function. It hands back a value and pauses, keeping its place until the next value is requested.

Calling count_up(3) does not run the body at all. It returns a generator object, and the body runs bit by bit as values are requested.

Seeing the laziness

Add prints to watch when the work happens:

The messages interleave: each yield pauses the generator, the loop body runs, and then the generator resumes.

Infinite sequences

Because values are made on demand, a generator can be endless, and you take only what you need.

islice(generator, n) takes the first n items. A normal list could never hold an infinite sequence.

Generator expressions

Like a list comprehension, but with round brackets. It builds nothing up front.

Pipelines

Generators can be chained so that data flows through, item by item, without intermediate lists.

This is how a program can process a huge log file line by line using almost no memory.

One-shot only

A generator can be used only once. After it is exhausted, it stays empty.

If you need to loop twice, create a new generator or store the results in a list.

Writing your own iterator class

Generators cover most needs. For completeness, a class becomes iterable by defining __iter__ and __next__:

The generator version is much shorter: def countdown(n): while n > 0: yield n; n -= 1.

Try it yourself

Write a generator chunks(items, size) that yields successive slices of a list with the given size, so chunks([1,2,3,4,5], 2) yields [1, 2], [3, 4], [5].

Summary

• A for loop calls iter() and then next() until StopIteration.
• A function with yield is a generator: it produces values lazily and pauses between them.
• Generator expressions use round brackets and never build the whole list.
• Chain generators into pipelines, and remember that a generator can be consumed only once.