We all know that asynchronous programming can greatly improve the throughput of the system for I/O-related programs, because in the process of reading and writing an I/O operation, the system can deal with other operations (usually other I/O operations) first, so how to implement asynchronous programming in Python?
The simple answer is that Python implements asynchronous programming through coroutine. So what exactly is a synergy? It will be a long story.
The story begins with yield (readers who are already familiar with yield can skip this section).
yield
Overview: yield is equivalent to return, which returns the corresponding value to the caller calling next() or send(), thereby giving CPU access. When the caller calls next() or send() again, it returns to the place where yield interrupts. If send has parameters, it returns the parameters to the variable assigned by yield, if not. You assign None as next(). But there's a problem here, that is, when nesting generators, the outer generators need to write a lot of code. Look at the following example:
Note that the following code runs debugging on Python 3.6
#!/usr/bin/env python
# encoding:utf-8
def inner_generator():
i = 0
while True:
i = yield i
if i > 10:
raise StopIteration
def outer_generator():
print("do something before yield")
from_inner = 0
from_outer = 1
g = inner_generator()
g.send(None)
while 1:
try:
from_inner = g.send(from_outer)
from_outer = yield from_inner
except StopIteration:
break
def main():
g = outer_generator()
g.send(None)
i = 0
while 1:
try:
i = g.send(i + 1)
print(i)
except StopIteration:
break
if __name__ == '__main__':
main()For simplification, yield from is introduced in Python 3.3
yield from
There are two advantages to using yield from.
1. The send parameter in main can be returned to the innermost generator all the time.
2. At the same time, we do not need to use while loop and send (), next() to iterate.
We can modify the above code as follows:
def inner_generator():
i = 0
while True:
i = yield i
if i > 10:
raise StopIteration
def outer_generator():
print("do something before coroutine start")
yield from inner_generator()
def main():
g = outer_generator()
g.send(None)
i = 0
while 1:
try:
i = g.send(i + 1)
print(i)
except StopIteration:
break
if __name__ == '__main__':
main()The results are as follows:
do something before coroutine start
1
2
3
4
5
6
7
8
9
10Here, the code snippet executed in inner_generator() can actually be considered a coroutine, so in general, the logic diagram is as follows: 
Next we'll see what the consortium looks like.
Coprocess coroutine
The concept of coroutines should evolve from processes and threads, which execute a piece of code independently, but the difference is that threads are lighter than processes and coroutines are lighter than threads. Multithreads are executed in the same process, and co-processes are usually executed in one thread. Their diagrams are as follows:
We all know that Python's threading efficiency is not high because of GIL(Global Interpreter Lock), and in * nix system, the cost of creating threads is not less than that of processes, so the efficiency of multi-threading is greatly restricted in concurrent operation. So it was later discovered that the execution of code fragments was interrupted by yield, and the use of cpu was surrendered, so the concept of protocol came into being. The concept of collaboration was formally introduced in Python 3.4. The code examples are as follows:
import asyncio
# Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html.
@asyncio.coroutine
def countdown(number, n):
while n > 0:
print('T-minus', n, '({})'.format(number))
yield from asyncio.sleep(1)
n -= 1
loop = asyncio.get_event_loop()
tasks = [
asyncio.ensure_future(countdown("A", 2)),
asyncio.ensure_future(countdown("B", 3))]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()The example shows the introduction of two important concepts, coprocess and event loop, in Python 3.4.
A collaboration is defined by the modifier @asyncio.coroutine, and all the collaboration tasks in tasks are executed by event loop. Later, a new async & await grammar was introduced in Python 3.5, which gave rise to the concept of native collaboration.
async & await
In Python 3.5, the aync & await grammar structure is introduced. Through "aync def", a coded fragment can be defined, which is similar to the @asyncio.coroutine modifier in Python 3.4, while await is equivalent to "yield from".
Let's start with a piece of code. This is a small program I wrote when I first started using async & await grammar.
#!/usr/bin/env python
# encoding:utf-8
import asyncio
import requests
import time
async def wait_download(url):
response = await requets.get(url)
print("get {} response complete.".format(url))
async def main():
start = time.time()
await asyncio.wait([
wait_download("http://www.163.com"),
wait_download("http://www.mi.com"),
wait_download("http://www.google.com")])
end = time.time()
print("Complete in {} seconds".format(end - start))
loop = asyncio.get_event_loop()
loop.run_until_complete(main())There will be such an error here:
Task exception was never retrieved
future: <Task finished coro=<wait_download() done, defined at asynctest.py:9> exception=TypeError("object Response can't be used in 'await' expression",)>
Traceback (most recent call last):
File "asynctest.py", line 10, in wait_download
data = await requests.get(url)
TypeError: object Response can't be used in 'await' expressionThis is because the Response object returned by the requests.get() function can't be used for await expressions, but if it can't be used for await, how can we achieve asynchrony?
The original Python await expression is something like "yield from", but await does parameter checking. It requires that the object in the await expression must be awaitable. So what is awaitable? The awaitable object must satisfy one of the following conditions:
1,A native coroutine object returned from a native coroutine function .
Native Cooperative Object
2,A generator-based coroutine object returned from a function decorated with types.coroutine() .
Type. coroutine () modifies the Generator-Based program object, noting that it is not asyncio.coroutine in Python 3.4
3,An object with an await method returning an iterator.
The await method is implemented and the iterator object is returned therein
Based on these conditional definitions, we can modify the code as follows:
#!/usr/bin/env python
# encoding:utf-8
import asyncio
import requests
import time
async def download(url): # A function defined by async def is a native coprocess object
response = requests.get(url)
print(response.text)
async def wait_download(url):
await download(url) # Here download(url) is a native collaboration object
print("get {} data complete.".format(url))
async def main():
start = time.time()
await asyncio.wait([
wait_download("http://www.163.com"),
wait_download("http://www.mi.com"),
wait_download("http://www.google.com")])
end = time.time()
print("Complete in {} seconds".format(end - start))
loop = asyncio.get_event_loop()
loop.run_until_complete(main())Well, now a real asynchronous programming program has finally come into being.
At present, the more powerful asynchronization is using uvloop or pyuv. These two latest Python libuv libuv libraries can provide more efficient event loop.
uvloop and pyuv
pyuv implements Python 2.x and 3.x, but the project has not been updated on github for a long time. I don't know if anyone else is maintaining it.
uvloop only implements 3.x, but the project is always active on github.
They are also very simple to use. Take uvloop for example, just add the following code
import asyncio
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())This is the end of Python asynchronous programming, and the introduction to this article is actually about the amazing performance created by the combination of Sanic and uvloop on the Internet. Interested students can find relevant articles. Maybe later I will write a special article on this topic. Welcome to exchange!