Python 3 uses asyncio module asyncio to realize concurrency of coroutines

Posted by daloss on Tue, 08 Oct 2019 02:38:44 +0200

Python 3 solves many problems of Python 2, such as GIL (Global Interpreter Lock), and introduces many powerful libraries as native libraries. Among them, asyncio libraries are a bright spot, which can be said to be the libraries Python 3 must understand.

Understand the asyncio library: https://www.cnblogs.com/zhaof/p/8490045.html

Learn about yield and await: https://zhuanlan.zhihu.com/p/27258289

Understand the event cycle: http://www.ruanyifeng.com/blog/2013/10/event_loop.html

                            https://www.cnblogs.com/saolv/p/9969936.html

Some pits in Python: https://www.zhihu.com/question/23474039

Python performance optimization: https://www.jianshu.com/p/9cfa1dc99769

The following is an example of using asyncio module asyncio to achieve concurrency of concurrent processes for future reference.

import time
import asyncio
from threading import Thread


async def do_task(params):  # Perform a single task
    print(time.time(), params)
    await asyncio.sleep(1)  # Note that you can't use time.sleep(), otherwise the whole world will sleep.
    print(time.time(), params)
    return params


async def do_task_group(params_group):  # Perform multiple tasks
    tasks = [asyncio.ensure_future(do_task(params), loop=io_loop) for params in params_group]
    results = await asyncio.gather(*tasks, loop=io_loop, return_exceptions=True)
    return results


async def do_task_groups(all_params, send_step=5):  # Executing multiple task groups
    # Divide all tasks into multiple task groups according to step size
    params_groups = [all_params[index: index + send_step] for index in range(0, len(all_params), send_step)]
    tasks = [asyncio.ensure_future(do_task_group(params_group), loop=io_loop) for params_group in params_groups]

    # Get and merge task results
    the_results = await asyncio.gather(*tasks, loop=io_loop, return_exceptions=True)
    results = []
    for result in the_results:
        results.extend(result)
    return results


def do_all_tasks(all_params):  # Perform all tasks
    results = asyncio.run_coroutine_threadsafe(do_task_groups(all_params), io_loop)
    print('Start all task in', time.time())
    return results.result()  # Blocking access to results, where transactions are actually blocked


def start_loop():  # Start the event loop
    io_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(io_loop)
    thread = Thread(target=io_loop.run_forever)
    thread.daemon = True
    thread.start()
    return io_loop

if __name__ == '__main__':
    io_loop = start_loop()
    all_params = list(range(10))
    print(do_all_tasks(all_params))

 

Topics: Programming Python