Python Iterators

Python Iterable

In Python, anything that you can loop over is called iterable. For example, list, strings, tuple, set, etc.

For an object to be considered an iterable, it must have the __iter()__ method.

numbers = [1, 3, 5, 7]

# find all the methods inside the numbers list
print(dir(numbers))

Output

['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

In the above code, we have called the dir() function on the numbers list. This function returns all the available methods inside the list object.

Among the output, you can see the __iter__ method. Let's see what happens if we call this method.

numbers = [1, 3, 5, 7]

# call the __iter__() method of the list
iter_value = numbers.__iter__()
print(iter_value)

Output

<list_iterator object at 0x7f2464c2c4c0>

Here, you can see the __iter__() method returns an iterator object.

Python Iterators

Iterator in Python is simply an object that allows us to iterate over data and return one at a time. For an object to be an iterator, it must implement two methods:

• __iter__()
• __next__()

They are also called the iterator protocol.

The next() Method

While iterating an object, the __next__() method returns the next value in the iteration. For example,

numbers = [2, 4, 6, 8]

# get the iterator
iter_value = numbers.__iter__()

# call the next method
item1 = iter_value.__next__()
print(item1)

Output

2

Initially, the __next__() method returns the first element of the list. Now, if we run the next method again, it should return the next item, which is 4.

numbers = [2, 4, 6, 8]

# get the iterator
iter_value = numbers.__iter__()

# call the next method
item1 = iter_value.__next__()
print(item1)

# access the next item
item2 = iter_value.__next__()
print(item2)

# access the next item
item3 = iter_value.__next__()
print(item3)

# access the next item
item4 = iter_value.__next__()
print(item4)

Output

2
4
6
8

Here, you can see each time we are calling the __next__() method, it is returning the next item in the list.

Using iter() and next() instead of __iter__() and __next__()

Instead of calling these special methods with an underscore, Python provides an elegant way to call __iter__() simply with the iter() function and __next__() with the next() function. For example,

numbers = [2, 4, 6, 8]

# get the iterator
iter_value = iter(numbers)

# call the next method
item1 = next(iter_value)
print(item1)

# access the next item
item2 = next(iter_value)
print(item2)

# access the next item
item3 = next(iter_value)
print(item3)

# access the next item
item4 = next(iter_value)
print(item4)

Output

2
4
6
8

Here, we have used

• iter(numbers) in place of numbers.__iter__()
• next(iter_value) in place of iter_value.__next__()

The above list has 4 items, so after using the next() method for 4 times, we are at the end of our list. Now, let's see what happens if we further try to get the next value.

numbers = [2, 4, 6, 8]

# get the iterator
iter_value = iter(numbers)

# call the next method
item1 = next(iter_value)
print(item1)

# access the next item
item2 = next(iter_value)
print(item2)

# access the next item
item3 = next(iter_value)
print(item3)

# access the next item
item4 = next(iter_value)
print(item4)

# access the next item
item5 = next(iter_value)
print(item5)

Output

2
4
6
8
Traceback (most recent call last):
  File "<string>", line 23, in <module>
StopIteration

Here, we are getting the StopIteration exception. This is because our list only has 4 elements and we are calling the next() method 5 times to access 5 elements.

To know about exceptions, please visit Python Exceptions.

Internal Working of for Loop using Iterator

Did you know that the for loops internally use the while loop and iterator to loop over sequences?

Let me show you an example. Suppose we have a for loop that iterates over a list.

num_list = [2, 3, 5, 7, 11]

# for loop to iterate through list
for element in num_list:
    print(element)

Output

2
3
5
7
11

Let's now see how this loop works internally.

num_list = [2, 3, 5, 7, 11]

# create an iterator object
iter_obj = iter(num_list)

# loop is always true
while True:
    try:
        # access each element of list using next()
        element = next(iter_obj)
        print(element)
    
    # if the next() method reaches the end
    # it will throw the exception 
    except StopIteration:
        break

Output

2
3
5
7
11

Here's how this code works:

• First, we have created an iterator object from a list using iter_obj = iter(num_list).
• Then, we run an infinite while loop.
• Inside the loop, we have used the next() method to return the next element in the list element = next(iter_obj) and print it. We have put this code inside the try block.
• When all the items of the list are iterated, the try block raises the StopIteration exception, and the except block catches it and breaks the loop.

This is how for loops work behind the scene. A for loop internally creates an iterator object and iterates over it, calling the next() method until a StopIteration exception is encountered.

Creating Custom Iterators in Python

Python also allows us to make our own iterator object to generate a sequence. For example, let's generate a sequence of even numbers such as 2, 4, 6, 8, and so on.

class Even:
    def __init__(self, max):
        self.n = 2
        self.max = max
        
    def __iter__(self):
        return self
        
    # customize the next() method to return only even numbers
    def __next__(self):
        if self.n <= self.max:
            result = self.n
            self.n += 2
            return result
            
        else:
            raise StopIteration
            

numbers = Even(10)

print(next(numbers))
print(next(numbers))
print(next(numbers))
print(next(numbers))

Output

2
4
6
8

In the above example, we have defined the __iter__() and __next__() method inside the Even class. To understand the working of the class, visit Python Class.

Here, we have modified the __next__() method to only return the even numbers as the next element (self.n += 2).

Use of Iterators

Iterators can be so useful when dealing with a large stream of data. For example, if we regularly use lists to store values, there is a chance that our computer would run out of memory.

However, with iterators, we can save resources as they return only one element at a time. So, in theory, we can deal with infinite data in finite memory.

Recommended Reading: Python Generators