Python process pool multiprocessing Pool - Python zero basics tutorial

Posted by jpr on Sat, 22 Jan 2022 12:02:45 +0100

catalogue

I Python process pool multiprocessing Introduction to pool
II Python process pool multiprocessing Pool usage
III Guess you like it

Zero basic Python learning route recommendation: Python learning directory >> Getting started with Python Basics

Python process Pool As explained earlier **Python thread pool** Similarly, although using multiple processes can improve efficiency, the creation of processes will consume a lot of computer resources( Process The creation of thread is much larger than the resources occupied by thread creation). Thread is the smallest running unit * * of the computer. All connection processes need to use thread pool. Why don't processes use process pool?

It should be noted that if you want to use the process module on Windows, you must write the code about the process in if __name__ == '__main__' Otherwise, using the process module under Windows will produce an exception. Not required under Unix / Linux.

I Python process pool multiprocessing Introduction to pool

The Pool class can provide a specified number of processes for users to call. When a new request is submitted to the Pool, if the Pool is not full, a new process will be created to execute the request. If the Pool is full, the request will be told to wait first, and a new process will not be created to execute these requests until there are processes in the Pool.

# Import process module

import multiprocessing



# Up to 3 processes are allowed to run at the same time

pool = multiprocessing.Pool(processes = 3)

1. apply - this function is used to pass indefinite parameters. The main process will be blocked until the end of function execution (not recommended and will not appear after 3.x). The function prototype is as follows:

apply(func, args=(), kwds={})

**2,apply_async * * - the same usage as apply, but it is non blocking and supports callback after the result is returned. The function prototype is as follows:

apply_async(func[, args=()[, kwds={}[, callback=None]]])

3. Map - the map method in the Pool class is basically the same as the built-in map function. It will block the process until the result is returned. The function prototype is as follows:

map(func, iterable, chunksize=None)

Note: Although the second parameter is an iterator, in practice, the program will not run the subprocess until the whole queue is ready.

**4,map_async * * - consistent with map usage, but it is non blocking. See apply for relevant matters_ Async, the function prototype is as follows:

map_async(func, iterable, chunksize, callback)

5. Close - close the process pool so that it no longer accepts new tasks.

6. terminal - end the work process, not processing unprocessed tasks.

7. Join - the main process blocks and waits for the child process to exit. The join method should be used after close or terminate.

II Python process pool multiprocessing Pool usage

# !usr/bin/env python
# -*- coding:utf-8 _*-
"""
@Author:Ape programming
@Blog(Personal blog address): www.codersrc.com
@File:Python Process pool multiprocessing.Pool.py
@Time:2021/05/09 07:37
@Motto:No small steps can lead to thousands of miles, no small streams can lead to rivers and seas. The brilliance of program life needs to be accumulated unremittingly!

"""


import multiprocessing
import time


def func(msg):
    print("in:", msg)
    time.sleep(3)
    print("out,", msg)

if __name__ == "__main__":
    # Here, the number of processes allowed to run at the same time should consider the number of machine CPUs. The number of processes should not be less than the number of CPUs,
    # Because even if it is larger than the number of CPUs, the task scheduling time is increased, but the efficiency can not be effectively improved
    pool = multiprocessing.Pool(processes = 3)
    item_list = ['processes1' ,'processes2' ,'processes3' ,'processes4' ,'processes5' ,]
    count = len(item_list)
    for item in item_list:
        msg = "python course %s" %item
        # The total number of processes to be executed is processes. When a process is completed, a new process will be added
        pool.apply_async(func, (msg,))

    pool.close()
    pool.join()  # Call the close function before calling join, otherwise an error will occur. After the close is executed, no new process will be added to the pool. The join function waits for all child processes to finish

'''
Output results:

in: python course processes1
in: python course processes2
in: python course processes3
out, python course processes1
in: python course processes4
out, python course processes2
in: python course processes5
out, python course processes3
out, python course processes4
out, python course processes5
'''

Code analysis:

5 tasks and 3 processes. Since it is allowed to execute up to 3 processes at the same time when the process pool is constructed, task 1 / task 2 / task 3 can be executed at the same time. According to the output results of heavy code, after task 1 / task 2 / task 3 is executed, the for loop enters the blocking state, Do not continue to execute task 4 / task 5 until one of task 1 / task 2 / task 3 is completed, and ensure that only 3 tasks can be executed at the same time( Process pool multiprocessing Pool and Thread pool ThreadPoolExecutor Same principle)

III Guess you like it

No reprint without permission: Ape programming » Python process pool multiprocessing Pool

[like (1)](javascript: 😉 [reward] (javascript: 😉

Topics: Python

Programmer Think

Python process pool multiprocessing Pool - Python zero basics tutorial

I Python process pool multiprocessing Introduction to pool

II Python process pool multiprocessing Pool usage

III Guess you like it

Hot Topics