asyncio asynchronous programming

Official documents:

• Chinese version: https://docs.python.org/zh-cn/3.8/library/asyncio.html
• English version: https://docs.python.org/3.8/library/asyncio.html

1. Event cycle

Event loop means that every time the main thread empties the tasks in the execution sequence, it goes to the event queue to check whether there are tasks waiting to be executed. If so, it takes out one task at a time and pushes it into the execution sequence for execution. This process is a cycle.

Understand it as an endless loop to detect and execute some code.

Event loop is the core of every asyncio application. Event loops run asynchronous tasks and callbacks, perform network IO operations, and run subprocesses.

Application developers should usually use high-level asyncio functions, such as asyncio.run() , there should be little need to reference a circular object or call its methods.

# Pseudo code

task list = [ Task 1, Task 2, Task 3,... ]

while True:
    Executable task list, completed task list = Go to the task list to check all tasks, and'Executable'and'Completed'Task return
    
    for Ready task in List of tasks ready:
        Perform ready tasks
    
    for Completed tasks in List of completed tasks:
        Remove completed tasks from the task list
    
    If all the tasks in the task list have been completed, the cycle is terminated
import asyncio

# To generate or get an event loop
loop = asyncio.get_event_loop()

# Put tasks in 'task list'
loop.run_until_complete(task)

2. Detailed explanation

2.1 async

Synergetic function, async def function name when defining function

Coprocessor object, the coprocessor object obtained by executing the coprocessor function ()

async def func():  # Define a coprocessor function
    pass

result = func()   # Get the coroutine object (at this time, the internal code of the function will not be executed)

Note: the internal code of the coroutine function can be executed only by adding the event loop, that is, the coroutine object is handed over to the event loop for processing

import asyncio

async def func():
    print('Life is short, I use it python')
   
result = func()

# loop = asyncio.get_event_loop()
# loop.run_until_complete(result)
asyncio.run(result)  # Equivalent to the above two lines, python 3 Convenient way to introduce after 7

2.2 await

await + wait object (collaboration object, Future, Task object - > IO wait)

2.2.1 coordination object

The coroutine object is the coroutine object obtained by executing the coroutine function ().

Example 1:

import asyncio

async def func():
    print('Life is short,')
    response = await asyncio.sleep(2)
    print('I use python', response)
    
asyncio.run(func()).

# result:
"""
Life is short,
I use python None
"""

Example 2:

import asyncio


async def others():
    print('Life is short,')
    await asyncio.sleep(2)
    print('I use python')
    return 'Return value'


async def func():
    print('Internal code of execution coprocessor function--->')

    # When an IO operation suspends the current collaboration task, continue to execute it after the IO operation is completed. When the current collaboration is suspended, the event loop can execute other collaboration tasks
    response = await others()

    print('IO The request ended with:', response)


asyncio.run(func())

"""
Internal code of execution coprocessor function--->
Life is short,
I use python
IO The request ends, and the result is: return value
"""

Example 3:

import asyncio


async def others():
    print('Life is short,')
    await asyncio.sleep(2)
    print('I use python')
    return 'Return value'


async def func():
    print('Internal code of execution coprocessor function--->')

    # When an IO operation suspends the current collaboration task, continue to execute it after the IO operation is completed. When the current collaboration is suspended, the event loop can execute other collaboration tasks
    response1 = await others()
    print('IO The request ended with:', response1)

    response2 = await others()
    print('IO The request ended with:', response2)


asyncio.run(func())
"""
Internal code of execution coprocessor function--->
Life is short,
I use python
IO The request ends, and the result is: return value
 Life is short,
I use python
IO The request ends, and the result is: return value
"""

await is to wait for the value of the object to get the result and then continue to execute.

2.2.2 Task object

Add multiple tasks to the event loop.

Tasks is used for concurrent scheduling of collaborative processes through asyncio create_ Create a task object in the form of task (collaboration object), so that the collaboration can be added to the event loop and wait for the scheduled execution. In addition to using asyncio create_ In addition to the task () function, you can also use the low-level loop create_ Task () or ensure_ The future() function. Manual instantiation of task objects is not recommended.

In essence, it encapsulates the collaboration object into a task object, immediately adds the collaboration into the event loop, and tracks the status of the collaboration at the same time.

Note: asyncio create_ The task () function was added in Python 3.7. Before Python 3.7, you can use the low-level asyncio ensure_ Future() function

Example 1:

import asyncio

async def func():
    print(1)
    await asyncio.sleep(2)
    print(2)
    return "Return value"

async def main():
    print("main start")
    # Create a collaboration process, encapsulate the collaboration process into a Task object, and immediately add it to the Task list of the event cycle, waiting for the event cycle to execute (the default is ready state).
    task1 = asyncio.create_task(func())
    # Create a collaboration process, encapsulate the collaboration process into a Task object, and immediately add it to the Task list of the event cycle, waiting for the event cycle to execute (the default is ready state).
    task2 = asyncio.create_task(func())
    print("main end")

    # When an IO operation is encountered during the execution of a collaboration, it will automatically switch to other tasks.
    # await here is to wait for the execution of the corresponding processes and obtain the results
    ret1 = await task1
    ret2 = await task2
    print(ret1, ret2)

asyncio.run(main())

"""
result:

main start
main end
1
1
2
2
 Return value return value
"""

Example 2 (simplified example 1):

import asyncio

async def func():
    print(1)
    await asyncio.sleep(2)
    print(2)
    return "Return value"

async def main():
    print("main start")
    # Create a collaboration process, encapsulate the collaboration process into the Task object and add it to the Task list of the event cycle, and wait for the event cycle to execute (the default is ready state).
    # In call
    task_list = [
        asyncio.create_task(func(), name="n1"),
        asyncio.create_task(func(), name="n2")
    ]
    print("main end")
    
    # When an IO operation is encountered during the execution of a collaboration, it will automatically switch to other tasks.
    # await here is to wait for all cooperation processes to be executed and save the return values of all cooperation processes to done
    # If the timeout value is set, it means that the maximum waiting time here is seconds. The returned value of the completed process is written to done. If it is not completed, it is written to pending.
    done, pending = await asyncio.wait(task_list, timeout=None)
    print(done, pending)

asyncio.run(main())

"""
result:

main start
main end
1
1
2
2
{<Task finished name='n1' coro=<func() done, defined at D:\coding\tronado_test\03_tasks.py:31> result='Return value'>, <Task finished name='n2' coro=<func() done, defined at D:\coding\tronado_test\03_tasks.py:31> result='Return value'>} set()
"""

Example 3:

import asyncio


async def func():
    print("Internal code of execution coprocessor function")
    # In case of IO operation, suspend the current collaboration (task), and continue to execute after the IO operation is completed. When the current process is suspended, the event loop can execute other processes (tasks).
    response = await asyncio.sleep(2)
    print("IO The request ended with:", response)

# Error: coroutine_list = [ asyncio.create_task(func()), asyncio.create_task(func()) ]
# You cannot directly asyncio here create_ Task, because the task is immediately added to the task list of the event cycle, but the event cycle has not been created at this time, an error will be reported.
# Therefore, you can add the collaboration object to the list to use.
coroutine_list = [
    func(),
    func()
]

# Use asyncio Wait encapsulates the list as a coroutine and calls asyncio The run implementation executes two coroutines
# asyncio.wait will execute ensure for each collaboration in the list_ Future, encapsulated as a Task object.
done,pending = asyncio.run( asyncio.wait(coroutine_list) )

"""
result:
Internal code of execution coprocessor function
 Internal code of execution coprocessor function
IO The request ended with: None
IO The request ended with: None
"""

2.2.3 async Future object

A Futureis a special low-level awaitable object that represents an eventual result of an asynchronous operation.

future is a special low-level waiting object that represents the end result of an asynchronous operation

Task inherits Future, and the processing of await results inside the task object is based on the Future object.

The Future object in asyncio is a relatively more bottom-level executable object. Generally, we will not use this object directly, but directly use the Task object to complete the Task and track the status. (Task is a subclass of Futrue)

Future provides us with the processing of the final result in asynchronous programming (the Task class also has the function of state processing).

Example 1:

import asyncio

async def main():
    # Get current event loop
    loop = asyncio.get_running_loop()
    # # Create a task (Future object) that does nothing.
    fut = loop.create_future()
    # Wait for the final result of the task (Future object). If there is no result, it will wait forever.
    await fut
    
asyncio.run(main())

# Result: will wait all the time

Example 2:

import asyncio

async def set_after(fut):
    await asyncio.sleep(2)
    fut.set_result("666")

async def main():
    # Get current event loop
    loop = asyncio.get_running_loop()

    # Create a task (Future object) without binding any behavior, then the task will never know when to end.
    fut = loop.create_future()

    # Create a Task (Task object) and bind the set_after function, the function will assign a value to fut after 2s.
    # That is, manually set the final result of the future task, and then the future can end.
    await loop.create_task(set_after(fut))

    # Wait for the Future object to get the final result, otherwise wait forever
    data = await fut
    print(data)

asyncio.run(main())

# Result: 666

The Future object itself is bound by the function, so you need to set it manually if you want the event to cycle to obtain the results of Future. The Task object inherits the Future object, which actually extends the Future. It can automatically execute set after the execution of the corresponding bound function_ Result to achieve automatic end.

Although Task objects are usually used, the essence of processing results is based on Future objects.

2.3 running asyncio program

asyncio.run(*coro*, ***, *debug=False*)

• Execute coroutine coro and return the result.

• This function will run the incoming coroutine, manage the asyncio event loop, terminate the asynchronous generator, and close the thread pool.

• This function cannot be called when there are other asyncio event loops running in the same thread.

• If debug is True, the event loop will run in debug mode.

• This function always creates a new event loop and closes at the end. It should be used as the main entry point for asyncio programs and should ideally be called only once.

import asyncio

async def main():
    await asyncio.sleep(1)
    print('hello')

asyncio.run(main())

# Result: hello

2.4 create task

asyncio.create_task(coro, *, name=None)

• Encapsulate the coro process into a Task and schedule its execution. Return the Task object.

• name is not None, it will use Task.set_name() To set the name of the task.

• The task will be get_running_loop() Executed in the loop returned. If the current thread is not running a loop, it will be thrown RuntimeError.

• This function was added in Python 3.7. Before Python 3.7, you can use low-level asyncio.ensure_future() Function.

import asyncio

async def func():
    print(1)
    await asyncio.sleep(2)
    print(2)
    return "Return value"

async def main():
    print("main start")
    # Create a collaboration process, encapsulate the collaboration process into the Task object and add it to the Task list of the event cycle, and wait for the event cycle to execute (the default is ready state).
    # In call
    task_list = [
        asyncio.create_task(func(), name="n1"),
        asyncio.create_task(func(), name="n2")
    ]
    print("main end")
    
    # When an IO operation is encountered during the execution of a collaboration, it will automatically switch to other tasks.
    # await here is to wait for all cooperation processes to be executed and save the return values of all cooperation processes to done
    # If the timeout value is set, it means that the maximum waiting time here is seconds. The returned value of the completed process is written to done. If it is not completed, it is written to pending.
    done, pending = await asyncio.wait(task_list, timeout=None)
    print(done, pending)

asyncio.run(main())

"""
result:

main start
main end
1
1
2
2
{<Task finished name='n1' coro=<func() done, defined at D:\coding\tronado_test\03_tasks.py:31> result='Return value'>, <Task finished name='n2' coro=<func() done, defined at D:\coding\tronado_test\03_tasks.py:31> result='Return value'>} set()
"""

2.5 sleep

coroutine asyncio.sleep(delay, result=None, *, loop=None)

• Blocks the number of seconds specified by delay.

• If result is specified, it will be returned to the caller when the collaboration is completed.

• sleep() always suspends the current task to allow other tasks to run.

• Setting delay to 0 provides an optimized path to allow other tasks to run. This can be used by long-running functions to avoid blocking the event loop throughout the function call.

Deprecated since version 3.8, will be removed in version 3.10: loop parameter.

Deprecated from version 3.8, version 3.10 will be deleted: loop parameter

The following collaboration example runs for 5 seconds and displays the current date once per second:

import asyncio
import datetime

async def display_date():
    loop = asyncio.get_running_loop()
    end_time = loop.time() + 5.0
    while True:
        print(datetime.datetime.now())
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(1)

asyncio.run(display_date())

"""
2021-06-16 09:45:57.199182
2021-06-16 09:45:58.199578
2021-06-16 09:45:59.200463
2021-06-16 09:46:00.200563
2021-06-16 09:46:01.201021
"""

3. Expansion

3.1 concurrent.futures.Future object

Objects used in asynchronous operation using thread pool and process pool.

Example:

import time
from concurrent.futures import Future
from concurrent.futures.thread import ThreadPoolExecutor
from concurrent.futures.process import ProcessPoolExecutor

def func(value):
    time.sleep(1)
    print('------>', value)

pool = ThreadPoolExecutor(max_workers=5)

# Or pool = ProcessPoolExecutor(max_workers=5)

for i in range(10):
    fut = pool.submit(func, i)
    print(fut)
    
"""
result:
<Future at 0x1f178d68730 state=running>
<Future at 0x1f1790dca00 state=running>
<Future at 0x1f1790dcd60 state=running>
<Future at 0x1f1790f7130 state=running>
<Future at 0x1f1790f74c0 state=running>
<Future at 0x1f1790f7850 state=pending>
<Future at 0x1f1790f7970 state=pending>
<Future at 0x1f1790f7a90 state=pending>
<Future at 0x1f1790f7bb0 state=pending>
<Future at 0x1f1790f7cd0 state=pending>
------>------>  01

------> 3------>
 ------>4
 2
------> ------>6 
5
------> ------>------> 8
 7
9
"""

There may be cross use of the above two future s, so here's a comparison.

Usage scenario: the use of collaborative process must be supported by a third-party library. Crawlers use asyncIO and aiohttp library. They can use collaborative process for asynchronous programming. However, when using some third-party libraries that do not support collaborative process, they need to use process pool and thread pool to realize asynchronous programming.

import time
import asyncio
import concurrent.futures

def func1():
    # A time-consuming operation
    time.sleep(2)
    return "SB"

async def main():
    loop = asyncio.get_running_loop()

    # 1. Run in the default loop's executor
    # Step 1: the internal will first call the submit method of ThreadPoolExecutor to apply for a thread in the thread pool to execute func1 function and return a concurrent futures. Future object
    # Step 2: call asyncio wrap_ Future will be concurrent futures. The future object is wrapped as asycio Future object.
    # Because concurrent futures. Future objects do not support await syntax, so they need to be wrapped as asycio Only future objects can be used.
    fut = loop.run_in_executor(None, func1)
    result = await fut
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    # with concurrent.futures.ThreadPoolExecutor() as pool:
    #     result = await loop.run_in_executor(
    #         pool, func1)
    #     print('custom thread pool', result)

    # 3. Run in a custom process pool:
    # with concurrent.futures.ProcessPoolExecutor() as pool:
    #     result = await loop.run_in_executor(
    #         pool, func1)
    #     print('custom process pool', result)

asyncio.run(main())

# Result: default thread pool SB

Example 2 (async + modules that do not support asynchronous programming):

import asyncio
import requests


async def download_image(url):
    # Send network request and download pictures (in case of IO request for downloading pictures from the network, switch to other tasks automatically)
    print("Start downloading:", url)
    loop = asyncio.get_event_loop()
    # The requests module does not support asynchronous operation by default, so it uses thread pool to cooperate with the implementation.
    future = loop.run_in_executor(None, requests.get, url)
    response = await future
    print('Download complete')
    # Save picture to local file
    file_name = url.rsplit('_')[-1]
    with open(file_name, mode='wb') as file_object:
        file_object.write(response.content)

if __name__ == '__main__':
    url_list = [
        'https://www3.autoimg.cn/newsdfs/g26/M02/35/A9/120x90_0_autohomecar__ChsEe12AXQ6AOOH_AAFocMs8nzU621.jpg',   'https://www2.autoimg.cn/newsdfs/g30/M01/3C/E2/120x90_0_autohomecar__ChcCSV2BBICAUntfAADjJFd6800429.jpg',    'https://www3.autoimg.cn/newsdfs/g26/M0B/3C/65/120x90_0_autohomecar__ChcCP12BFCmAIO83AAGq7vK0sGY193.jpg'
    ]
    
    tasks = [download_image(url) for url in url_list]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(tasks))
    
"""
Result: start downloading: https://www3.autoimg.cn/newsdfs/g26/M0B/3C/65/120x90_0_autohomecar__ChcCP12BFCmAIO83AAGq7vK0sGY193.jpg
 Start downloading: https://www2.autoimg.cn/newsdfs/g30/M01/3C/E2/120x90_0_autohomecar__ChcCSV2BBICAUntfAADjJFd6800429.jpg
 Start downloading: https://www3.autoimg.cn/newsdfs/g26/M02/35/A9/120x90_0_autohomecar__ChsEe12AXQ6AOOH_AAFocMs8nzU621.jpg
 Download complete
 Download complete
 Download complete
"""

3.2 asynchronous iterators

What is an asynchronous iterator

Realized aiter() and anext() Method. anext must return a awaitable Object. async for Will handle asynchronous iterators anext() Method until it raises a StopAsyncIteration Abnormal. from PEP 492 introduce.

What is an asynchronous iteratable object?

Can be in async for The object used in the statement. Must pass its aiter() Method returns a asynchronous iterator . from PEP 492 introduce.

Asynchronous iterators don't really do much. They just support async for syntax.

import asyncio


class Reader(object):
    """ Custom asynchronous iterators (also asynchronous iteratable objects) """
    def __init__(self):
        self.count = 0
    async def readline(self):
        # await asyncio.sleep(1)
        self.count += 1
        if self.count == 100:
            return None
        return self.count
    def __aiter__(self):
        return self
    async def __anext__(self):
        val = await self.readline()
        if val == None:
            raise StopAsyncIteration
        return val
     
async def func():
    # Create asynchronous iteratable objects
    async_iter = Reader()
    # async for must be placed in the async def function, otherwise the syntax is incorrect.
    async for item in async_iter:
        print(item)
         
         
asyncio.run(func())

3.3 asynchronous context manager

Such objects are defined by aenter() and aexit() Method to async with Statement. from PEP 492 introduce.

import asyncio


class AsyncContextManager:
    def __init__(self):
        self.conn = None

    async def do_something(self):
        # Asynchronous operation database
        return 666

    async def __aenter__(self):
        # Asynchronously linked database
        self.conn = await asyncio.sleep(1)
        return self

    async def __aexit__(self, exc_type, exc, tb):
        # Close database links asynchronously
        await asyncio.sleep(1)


async def func():
    async with AsyncContextManager() as f:
        result = await f.do_something()
        print(result)


asyncio.run(func())

# Result: 666

3.4 uvloop

Is an alternative to the asyncio event loop, which is more efficient than the default asyncio event loop.

Installation:

pip install uvloop

The following codes must be added before use:

import asyncio
import uvloop
# Replace the default event scheme of asyncio with uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

# Write asyncio code, which is consistent with the code written before.
# The internal event loop automation becomes uvloop
asyncio.run(...)