1 Introduction to generator
Building in Python iterator You must implement a class with iter() and next() methods to track the internal state and raise StopIteration when there is no return value.
The Python generator is an easy way to create iterators.
Simply put, a generator is a function that returns an object (iterator) that can be iterated (one value at a time).
2 create generator
Creating generators in Python is fairly simple. It is as simple as defining a normal function, but uses a yield statement instead of a return statement.
If a function contains at least one yield statement (it may contain other yield or return statements), it becomes a generator function.
Both yield and return will return some values from the function. The difference is that the return statement completely terminates the function, while the yield statement pauses the function, saves all its state, and then continues execution in subsequent calls.
3 difference between generator function and ordinary function
Here are the differences between generator functions and ordinary functions:
- The generator function contains one or more yield statements;
- When called, it returns an object (iterator), but execution does not begin immediately;
- Methods like iter() and next() are implemented automatically. Therefore, you can use next() to iterate over these items;
- Once the function yield, the function will pause and transfer control to the caller;
- Local variables and their states are remembered between successive calls;
- Finally, when the function terminates, StopIteration is automatically raised on further calls.
The following is an example to illustrate all the points mentioned above. There is one named my_ Generator function of Gen (), which has several yield statements.
# A simple generator function
def my_gen():
n = 1
print('This is printed first')
# Generator function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n
function:
>>> # It returns an object but does not start execution immediately. >>> a = my_gen() >>> # We can iterate through the items using next(). >>> next(a) This is printed first 1 >>> # Once the function yields, the function is paused and the control is transferred to the caller. >>> # Local variables and theirs states are remembered between successive calls. >>> next(a) This is printed second 2 >>> next(a) This is printed at last 3 >>> # Finally, when the function terminates, StopIteration is raised automatically on further calls. >>> next(a) Traceback (most recent call last): ... StopIteration >>> next(a) Traceback (most recent call last): ... StopIteration
In the above example, it is interesting to remember the value of variable n between each call.
This is different from ordinary functions. When the generating function is generated, the local variables will not be destroyed. Moreover, the generator object can only be iterated once.
To restart the process, you need to use a = my_ Something like Gen () creates another generator object.
Finally, note that you can use the generator of the for loop directly.
This is because the for loop accepts an iterator and iterates over it using the next() function. When StopIteration is raised, it ends automatically.
# A simple generator function
def my_gen():
n = 1
print('This is printed first')
# Generator function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n
# Using for loop
for item in my_gen():
print(item)
Output:
This is printed first 1 This is printed second 2 This is printed at last 3
4 generator with loop
Generally, the generator function is implemented with a loop with appropriate termination conditions.
Look at an example of a generator that can invert a string
def rev_str(my_str):
length = len(my_str)
for i in range(length - 1, -1, -1):
yield my_str[i]
# For loop to reverse the string
for char in rev_str("hello"):
print(char)
Output:
o l l e h
In this example, the range() function is used to get the index in reverse order using the for loop.
Note: this generator function can handle not only strings, but also other types of iteratable objects, such as list, tuple, etc.
5 generator expression
Using generator expressions, you can easily create simple generators dynamically. It makes it easy to build generators.
Similar to lambda functions that create anonymous functions, generator expressions create anonymous generator functions.
The syntax of the generator expression is similar to the list derivation in Python. But the square brackets were replaced by parentheses.
The main differences between list derivation and generator expression are:
- The list derivation generates the entire list, while the generator expression generates one item at a time.
- Generator expressions have lazy execution (items are generated only when requested).
Therefore, the generator expression is much more memory efficient than the equivalent list derivation.
# Initialize the list my_list = [1, 3, 6, 10] # square each term using list comprehension list_ = [x**2 for x in my_list] # same thing can be done using a generator expression # generator expressions are surrounded by parenthesis () generator = (x**2 for x in my_list) print(list_) print(generator)
Output:
[1, 9, 36, 100] <generator object <genexpr> at 0x7f5d4eb4bf50>
As you can see above, the generator expression does not immediately produce the desired result. Instead, it returns a generator object that generates items only as needed.
Here is how to get items from the generator:
# Initialize the list my_list = [1, 3, 6, 10] a = (x**2 for x in my_list) print(next(a)) print(next(a)) print(next(a)) print(next(a)) next(a)
Output:
1 9 36 100 Traceback (most recent call last): File "<string>", line 15, in <module> StopIteration
Generator expressions can be used as function parameters. When used in this way, parentheses can be omitted.
>>> sum(x**2 for x in my_list) 146 >>> max(x**2 for x in my_list) 100
6 use of generator
Why generators are so powerful:
1. Easy to implement
Compared with iterator classes, generators can be implemented in a clear and concise way. The following is an example of implementing a power 2 sequence using an iterator class:
class PowTwo:
def __init__(self, max=0):
self.n = 0
self.max = max
def __iter__(self):
return self
def __next__(self):
if self.n > self.max:
raise StopIteration
result = 2 ** self.n
self.n += 1
return result
The above procedure is lengthy and confusing. Now, use a generator function to do the same thing.
def PowTwoGen(max=0):
n = 0
while n < max:
yield 2 ** n
n += 1
Because the generator can automatically track details, the implementation is more concise.
2. Save memory
A normal function that returns a sequence creates the entire sequence in memory before returning the result. If the number of entries in the sequence is very large, this is a bit excessive.
The generator implementation of this sequence is memory friendly and preferred because it produces only one item at a time.
3. Represents unlimited data flow
Generators are excellent media for representing infinite data streams. Infinite data streams cannot be stored in memory, and because generators can only produce one item at a time, they can represent infinite data streams.
The following generator function can generate all even numbers (at least in theory).
def all_even():
n = 0
while True:
yield n
n += 2
4. Pipe generator
Multiple generators can be used to pipeline a series of operations.
Suppose there is a generator that can generate numbers in the Fibonacci sequence. We have another square generator.
If you want to require the sum of squares of the numbers in the Fibonacci sequence, you can realize it by pipelining the output of the generator function.
def fibonacci_numbers(nums):
x, y = 0, 1
for _ in range(nums):
x, y = y, x+y
yield x
def square(nums):
for num in nums:
yield num**2
print(sum(square(fibonacci_numbers(10))))
Output:
4895
This pipeline is efficient and easy to read.
Reference: programiz