Iterators and Generators

Consuming Iterators

At a very high level,iterators are just a way to process items sequentially.There are various patterns which can be used when working with iterators and there is a python module itertools that has much more enhanced functionality.

Manually consuming an Iterator

Suppose you want to consume and iterable item like a file but with out using a loop and you want to control the exit of the iterable as well. For such situations we can use the next() function and handle the StopIteration exception.

For example:

with open('/path/my-file') as f:
    try:
        while True:
            line = next(f)
            print(line,end='')

    except StopIteration
        pass

Using this technique we have much more control on how we want to handle the iteration. Infact you can get hold of an iterator for a list:

items = [1,2,3]
it = iter(items)
next(it)

Iterators for custom container objects

We can define custom iterators for our custom container objects that internally holds a list,tuple or any other iterable.All we need to define is the iter method in our class which will delegate the iteration to the internally defined container. Example

class Node:
    def __init__(self,value):
        self._value = value
        self._children = list()

    def __repr__(self):
        return f'Node({self._value})'
    
    def add_child(self,node):
        self._children.append(node)

    def __iter__(self):
        return iter(self._children)

Usage

if __name__ == '__main__':
    root = Node(0)
    c1 = Node(1)
    c2 = Node(2)

    root.add_child(c1)
    root.add_child(c2)
    for ch in root:
        print(ch)
    #Outputs Node(1) and Node(2)

Defining new iterator patterns

We have several in built functions like range(),reversed(), we can also create our own iterator functions like these by making use of generators. For example suppose we want to create a generator that produces a range of floating-point numbers.

def frange(start,stop,increment):
    x = start
    while x < stop:
        yield x

        x += increment

usage

for n in frange(0,4,0.5):
    print(n)

The presence of yield turns this function into a generator.

Iterating in reverse

Suppose we want to iterate a sequence in reverse, we can use the reversed builtin function:

a = [2,4,6,8]
for x in reversed(a):
    print(a)

Reversed iteration only works if the object in question has a size that can be determined.

We can also include reversed iteration in user defined classes by implementing the __reversed__ function:

class Countdown:
    def __init__(self,start):
        self.start = start

    # Forward iterator
    def __iter__(self):
        n = self.start
        while n > 0:
            yield n
            n -=1

    # Reverse iterator
    def __reversed__(self):
        n = 1
        while n <= self.start:
            yield n
            n += 1

Some Iterator magic functions

slicing an iterable - iterator.islice()

This function is perfectly suited for taking slices of iterators and generators which cannot be normally sliced,because no information is known about their length.The result of islice() is an iterator that produces the desired slice items. Example:

import itertools
def count(n):
    while True:
        yield n
        n +=1

c = count(0)
for x in itertools.islice(c,10,20):
    print(x)

Skipping parts of an iterable - iterator.dropwhile()

To use the function we supply it with a function and an iterable. Example: Suppose we want to skip the initial comment lines in the passwd file

from itertools import dropwhile
with open('/etc/passwd') as f:
    for line in dropwhile(lambda line: line.startswith('#'),f):
        print(line,end='')

Iterating over all possible permutation and combination

For this the itertools module provides two functions:

1. itertools.permutations()

This takes in an iterable and produces a sequence of tuples that rearranges all of the items into all possible permutations: Example:

items = ['a','b','c','d']
from itertools import permutations

for p in permutations(items):
    print(p)

We can optionally provide a length argument

items = ['a','b','c','d']
from itertools import permutations

for p in permutations(items,2):
    print(p)

2. itertools.combinations()

This also takes in an iterable and an optional length argument Example:

items = ['a','b','c','d']
from itertools import combinations

for p in combinations(items):
    print(p)

There is a subtle difference between combinations and permutations,in combinations the order (‘a’,‘b’) is considered to be the same as (‘b’,‘a’) which is not produced.

Iterating over multiple sequences simultaneously

We can iterate over multiple sequences using zip() function.

Example:

xpts = [1,5,7,5,6]
ypts = [122,78,65,48,33]
for x, y in zip(xpts,ypts):
    print(x,y)

zip works by creating an iterator that produces tuples (x,y) where x is taken from xpts and y from ypts and stops whenever one of the input sequences is exhausted.Thus the length is the same as the shortest sequence. if this behaviour is not desired we can use the zip_longest() function.