Hiprup

What are generators in Python and how do they work?

A generator is a function that yields values one at a time instead of returning them all at once. It pauses execution at each yield and resumes from where it left off when the next value is requested.

How it works under the hood: the interpreter compiles defs that contain yield into generator-function objects. Each generator object holds its own stack frame, locals, and instruction pointer. yield suspends the frame and returns a value to the caller; next() restores the frame and continues. When the function returns (or falls off the end), the generator raises StopIteration, which is what ends a for loop.

Generator expressions — a one-liner form, nearly identical to a list comprehension but with parentheses:

# Generator function
def countdown(n):
    while n > 0:
        yield n      # Pause here, return value
        n -= 1       # Resume here on next call

for num in countdown(5):
    print(num)  # 5, 4, 3, 2, 1

# Memory-efficient file processing
def read_large_file(file_path):
    with open(file_path) as f:
        for line in f:
            yield line.strip()

# Process 10GB file with constant memory
for line in read_large_file('huge_log.txt'):
    if 'ERROR' in line:
        process_error(line)

# Infinite sequence
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
first_10 = [next(fib) for _ in range(10)]
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

# Generator pipeline (composable processing)
def parse_logs(lines):
    for line in lines:
        yield line.split(' | ')

def filter_errors(records):
    for record in records:
        if record[0] == 'ERROR':
            yield record

lines = read_large_file('app.log')
records = parse_logs(lines)
errors = filter_errors(records)
# Nothing executes until we iterate!
for error in errors:
    alert(error)

# Generator expression
sum_squares = sum(x**2 for x in range(1_000_000))  # No list in memory

countdown yields values and pauses between each. read_large_file processes a file line by line — constant memory regardless of file size. fibonacci generates infinite Fibonacci numbers — take only what you need. The pipeline chains three generators — nothing executes until the final for loop iterates.

Generator expressions avoid creating intermediate lists.

The file processing example (constant memory for huge files) is the strongest practical argument. Show the generator pipeline pattern (composable, lazy processing).

Know the difference between yield (pause) and return (terminate). Generator expressions vs list comprehensions for memory efficiency.

What are generators in Python and how do they work? | Hiprup