Python iterator, generator, decorator

Posted by mkubota on Mon, 07 Mar 2022 09:15:17 +0100


Generally speaking, the process of extracting data from an object in turn is called traversal, and this means is called iteration (repeatedly execute a certain code block, and take the result of each iteration as the initial value of the next iteration).
Iteratable object: refers to the object that can be used for... in... Loop, such as: set, list, ancestor, dictionary, string, iterator, etc.

  • In python, if an object implements__ iter__ Method, we call it iteratable object. You can view the internal implementation of set\list\tuple... And other source codes__ iter__ method

  • If an object is not implemented__ iter__ Method, but using for... in will throw TypeError: 'xxx' object is not iterable

  • You can use isinstance (obj, iteratable) to determine whether an object is an iteratable object. For example:

    from import Iterable
    # int a
    a = 1
    print(isinstance(a, Iterable))  # False
    # str b
    b = "lalalalala"
    print(isinstance(b, Iterable))  # True
    # set c
    c = set([1, 2])
    print(isinstance(c, Iterable))  # True
    # list d
    d = [1,2,3,"a"]
    print(isinstance(d, Iterable)) # True
    # dict e
    e = {"a":1,"b":2,"c":333}
    print(isinstance(e, Iterable)) # True
    # tuple f
    f = (1,3,4,"b","d",)
    print(isinstance(f, Iterable)) # We can also implement it ourselves__ iter__ To turn a class instance object into an iteratable object:
  • We can do it ourselves__ iter__ To turn a class instance object into an iteratable object:

    Implement the requirements of iterative objects by yourself
    1. In python, if an object is implemented at the same time__ iter__ And__ next__ (get the next value) method, then it is an iterator object.

    2. You can use the built-in function next(iterator) or instance object__ next__ () method to get the value of the current iteration

    3. Iterators must be iteratable objects, and iteratable objects are not necessarily iterators.

    4. If you continue to call next() after traversing the iteratable object, you will throw a StopIteration exception.

    from import Iterator, Iterable
    class MyIterator:
        def __init__(self, array_list):
            self.array_list = array_list
            self.index = 0
        def __iter__(self):
            return self
        def __next__(self):
            if self.index < len(self.array_list):
                val = self.array_list[self.index]
                self.index += 1
                return val
                raise StopIteration
    # If the parent class is an iterator, the child class will also be an iterator
    class MySubIterator(MyIterator):
        def __init__(self):
    myIterator = MyIterator([1, 2, 3, 4])
    # Judge whether it is an iterative object
    print(isinstance(myIterator, Iterable))  # True
    # Determine whether it is an iterator
    print(isinstance(myIterator, Iterator))  # True
    # Subclass instantiation
    mySubIterator = MySubIterator()
    print(isinstance(mySubIterator, Iterator))  # True
    # Iterate
    print(next(myIterator))  # 1
    print(myIterator.__next__())  # 2
    print(next(myIterator))  # 3
    print(next(myIterator))  # 4
    print(next(myIterator))  # raise StopIteration
  • Advantages and disadvantages of iterators:

    - advantage:
    	The iterator object represents a data flow that can be called when needed next To get a value; Therefore, it always keeps only one value in memory,
    	For small memory consumption, unlimited data streams can be stored.
    	Better than other containers, all elements need to be stored in memory at one time, such as lists, sets and dictionaries...Wait.
    - Disadvantages:
    	1.Cannot get the length of stored elements unless the count is complete.
    	2.The value is not flexible and can only be taken backwards, next()Always return the next value; Unable to fetch the specified value(Can't be like a dictionary key,Subscript of or list),Moreover, the life cycle of the iterator object is one-time, and the life cycle ends when the element is iterated.


Definition: in Python, the mechanism of calculating while looping is called generator; At the same time, the generator object is also an iterator object, so it has the characteristics of iterator;
For example, it supports for loop, next() method, etc
Function: the elements in the object are calculated according to some algorithm, and the subsequent elements are continuously calculated during the cycle, so there is no need to create a complete list, so as to save a lot of space. Simple generator: you can get a generator object by changing the list generator [] to ().

# List generation
_list = [i for i in range(10)]
print(type(_list))  # <class 'list'>
print(_list)  # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

# generator 
_generator = (i for i in range(10))
print(type(_generator))  # <class 'generator'>
print(_generator)  # <generator object <genexpr> at 0x7fbcd92c9ba0>

# Generator object value
print(_generator.__next__())  # 0
print(next(_generator)) # 1
# Start with the third element!
for x in _generator:
    print(x)  # 2,3,4,5,6,7,8,9

Because the generator object also has the characteristics of iterators, calling the next() method after the element iteration will cause StopIteration.
Function object generator: the return value of a function object with a yield statement is a generator object.

def gen_generator():
    yield 1
def func():
    return 1
print(gen_generator(), type(gen_generator()))  
# <generator object gen_generator at 0x7fe68b2c8b30> <class 'generator'>
print(func(), type(func()))  
# 1 <class 'int'>
def gen_generator():
    yield "start"
    for i in range(2):
        yield i
    yield "finish"

gen = gen_generator()
print("from gen Object",next(gen))
print("from gen Object",next(gen))
print("from gen Object",next(gen))
print("from gen Object",next(gen))

#Take the first value start from the gen object
#Take the second value 0 from the gen object
#Take the third value 1 from the gen object
#Take the fourth value finish from the gen object
# StopIteration
#print("take five values from the gen object", next(gen)) 

#It's equivalent to
#gen2 = (i for i in ["start",1,2,"finish"])

Note: yield returns only one element at a time. Even if the returned element is an iteratable object, it is returned at one time

def gen_generator2():
    yield [1, 2, 3]
s = gen_generator2()
print(next(s))  # [1, 2, 3]

Advanced application of yield generator: send() method, pass the value to yield and return it (it will be returned immediately!);

If None is passed, it is equivalent to next(generator).

def consumer():
    r = ''
    while True:
        n = yield r
        if not n:
        print(f'[CONSUMER] Consuming get params.. ({n})')
        if n == 3:
            r = '500 Error'
            r = '200 OK'
def produce(c):
    c.send(None)  # Start generator
    n = 0
    while n < 5:
        n = n + 1
        print(f'[PRODUCER] Producing with params.. ({n})')
        r = c.send(n)  # Once n has a value, switch to consumer for execution
        print(f'[PRODUCER] Consumer return : [{r}]')
        if not r.startswith('200'):
            print("If the service returned by the consumer is abnormal, the production will be ended and the consumer will be shut down")
            c.close()  # Close generator
consume = consumer()

[PRODUCER] Producing with params.. (1)
[CONSUMER] Consuming get params.. (1)
[PRODUCER] Consumer return : [200 OK]
[PRODUCER] Producing with params.. (2)
[CONSUMER] Consuming get params.. (2)
[PRODUCER] Consumer return : [200 OK]
[PRODUCER] Producing with params.. (3)
[CONSUMER] Consuming get params.. (3)
[PRODUCER] Consumer return : [500 Error]
If the service returned by the consumer is abnormal, the production will be ended and the consumer will be shut down

The basic function of yield from iterable syntax is to return a generator object and provide a "data transmission pipeline",
Yield from item is the abbreviation of for item in Item: yield item;
And the internal help us achieve a lot of exception handling, simplifying the coding complexity. yield cannot get the return value of generator return:

def my_generator2(n, end_case):
    for i in range(n):
        if i == end_case:
            return f'When i==`{i}`When, interrupt the program.'
            yield i
g = my_generator2(5, 2)  # call
    print(next(g))  # 0
    print(next(g))  # 1
    print(next(g))  # End to be triggered here_ Case
except StopIteration as exc:
    print(exc.value)  # When i==`2 ', interrupt the program.

Using yield from can be simplified to:

def my_generator3(n, end_case):
    for i in range(n):
        if i == end_case:
            return f'When i==`{i}`When the program is interrupted.'
            yield i
def wrap_my_generator(generator):  # Will my_ The return value of the generator is wrapped into a generator
    result = yield from generator
    yield result
g = my_generator3(5, 2)  # call
for _ in wrap_my_generator(g):
# Output:
# 0
# 1
# When i==`2 ', interrupt the program.
yield from There are the following conceptual nouns:
1,Caller: the client (caller) code that calls the delegate generator (in the previous section) wrap_my_generator(g))
2,Delegate generators: including yield from Generator function of expression(packing),The function is to provide a pipeline for data transmission (above) wrap_my_generator)
3,Sub generator: yield from Generator function object added later (above my_generator3 Instance object for g)
The caller interacts with the generator through this "wrapper function", namely "caller"——>Delegate generator——>Generator function
 Here is an example to help you understand

# Sub generator
def average_gen():
    total = 0
    count = 0
    average = 0
    while True:
        new_num = yield average
        if new_num is None:
        count += 1
        total += new_num
        average = total / count
    # Each return means the end of the current collaboration.
    return total, count, average

# Delegate generator
def proxy_gen():
    while True:
        # Only when the sub generator is about to return, the variable on the left of yield from will be assigned and the subsequent code will be executed.
        total, count, average = yield from average_gen()
        print("Total incoming {} Number of values, sum:{},average:{}".format(count, total, average))

# Caller
def main():
    calc_average = proxy_gen()
    next(calc_average)  # Activation process
    calc_average.send(10)  # Incoming: 10
    calc_average.send(None)  # send(None) is equal to next(calc_acerage), that is, it will go to average_ return statement in gen
    print("================== Reopen collaboration ===================")
    calc_average.send(20)  # Incoming: 20
    calc_average.send(30)  # Incoming: 30
    calc_average.send(None)  # End the process

if __name__ == '__main__':
# Output:
# A total of 1 value is passed in, total: 10, average: 10.0
# ==================Reopen collaboration===================
# A total of 2 values are passed in, total: 50, average: 25.0


In one sentence, decorators are nested calls to functions

It is mainly applied in three aspects:

  • Print program execution time
  • Collect program execution logs
  • Used for interface access authentication

Let's start with a simple example

def decorator_get_function_name(func):
    Get the name of the running function

    def wrapper(*arg):
        :param arg:
        print(f"Current running method name:{func.__name__}  with  params: {arg}")
        return func(*arg)

    return wrapper

# @func_name is the syntax sugar of python
def test_func_add(x, y):
    print(x + y)

def test_func_sub(x, y):
    print(x - y)

test_func_add(1, 2)
# Output:
# Current running method name: test_func_add  with  params: (1, 2)
# 3
# If you don't use grammar sugar, you can also use the following methods, and the effect is the same
decorator_get_function_name(test_func_sub)(3, 5)
# Remember the quotation mentioned earlier? We can also use another way to achieve the same 👆 Same effect
dec_obj = decorator_get_function_name(test_func_sub)  # This is equivalent to the wrapper object
dec_obj(3,5)  # This is equivalent to wrapper(3,5)
# Output:
# Current running method name: test_func_sub  with  params: (3, 5)
# -2

It is often used for authentication verification. For example, the author will use it for login verification:

def login_check(func):
    def wrapper(request, *args, **kwargs):
        if not request.session.get('login_status'):
            return HttpResponseRedirect('/api/login/')
        return func(request, *args, **kwargs)
    return wrapper

def edit_config():

Multiple decorator instances

def w1(func):
    def wrapper(*args, **kwargs):
        print("Here is the first verification")
        return func(*args, **kwargs)

    return wrapper

def w2(func):
    def wrapper(*args, **kwargs):
        print("Here is the second check")
        return func(*args, **kwargs)

    return wrapper

def w3(func):
    def wrapper(*args, **kwargs):
        print("Here is the third check")
        return func(*args, **kwargs)

    return wrapper

@w2  # This is actually w2(w1(f1))
@w1  # This is w1(f1)
def f1():
    print(f"i`m f1, at {f1}")


Here is the second check
 Here is the first verification
i`m f1, at <function f1 at 0x113fe83a0>

Note: when writing a decorator, you'd better add the wrap of functools before the implementation, which can retain the name and properties of the original function

#No wraps
def my_decorator(func):
    def wrapper(*args, **kwargs):
        print('Calling decorated function...')
        return func(*args, **kwargs)
    return wrapper

def example():
    print('Called example function')

print(example.__name__, example.__doc__)

# Add wraps
import functools

def my_decorator(func):
    def wrapper(*args, **kwargs):
        print('Calling decorated function...')
        return func(*args, **kwargs)
    return wrapper

def example():
    print('Called example function')

print(example.__name__, example.__doc__)

wrapper decoratord
example Docstring

Log printing case

from functools import wraps

# This is the decoration function
def logger(func):

    def wrapper(*args, **kw):
        print('I'm going to start the calculation:{} Function:'.format(func.__name__))

        # The real implementation is this line.
        func(*args, **kw)

        print('Aha, I've finished the calculation. Add yourself a chicken leg!!')
    return wrapper

def add(x, y):
    print('{} + {} = {}'.format(x, y, x+y))

add(200, 50)

I'm going to start the calculation: add Function:
200 + 50 = 250
 Aha, I've finished the calculation. Add yourself a chicken leg!!

Implementation of decorator with parameters

from functools import wraps

def say_hello(contry):

    def wrapper(func):
        def deco(*args, **kwargs):
            if contry == "china":
            elif contry == "america":

            # # Where the function is actually executed
            func(*args, **kwargs)
        return deco
    return wrapper

def chinese():
    print("I come from China.")

def american():
    print("I am from America.")


I come from China.
I am from America.

Decorator class

The above are decorators based on function implementation. When reading other people's code, you can often find decorators based on class implementation.

Based on the implementation of class decorator, call and__ init__ Two built-in functions.
init: receive decorated function
call: implement decoration logic.

1. Without parameters

class logger(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        print("[INFO]: the function {func}() is running..."\
        return self.func(*args, **kwargs)

def say(something):
    print("say {}!".format(something))


[INFO]: the function say() is running...
say hello!

2. Class with parameters

#Decorator with parameters

class logger(object):
    def __init__(self,level='INFO'):
        self.level = level

    def __call__(self,func):
        def wrapper(*args, **kwargs):
            print("[{level}]: the function {func}() is running..." \
                  .format(level=self.level, func=func.__name__))
        return wrapper

def say(something):
    print("say {}!".format(something))


[WARNING]: the function say() is running...
say hello!

Topics: Python