Multi process

A running program is called a process

3. Multi process and multi task

3.1 process creation

# 1. Import process package
import multiprocessing
# 2. Create process objects through process classes
 Process object = multiprocessing.Process()
# 3. Start the process and perform the task
 Process object.start()

3.2 creating process objects through process classes

Process object = multiprocessing Process (target = task name)

Parameter name	explain
target	The name of the target task to be executed. Here, it refers to the name of the gain / loss function (method name)
name	Process name, generally not set
group	Process group. Currently, only None can be used

3.3 process creation and startup code

# Create child process -- singing
sing_process = multiprocessing.Process(target=sing)
# Create child process -- dance
dance_process = multiprocessing.Process(target=dance)
# Start process
sing_process.start()
dance_process.start()

Single task py

import time

# sing
def sing():
	for i in range(3)
		print("sing...")
		time.sleep(0.5)

# dance
def dance()
	for i in range(3)
		print("dance...")
		time.sleep(0.5)

if __name__ = '__main__':
	sing()
	dance()

Multitasking using multiple processes py

# 1. Import process package
import multiprocessing
import time

# sing
def sing():
	for i in range(3)
		print("sing...")
		time.sleep(0.5)

# dance
def dance()
	for i in range(3)
		print("dance...")
		time.sleep(0.5)
        
if __name__ = '__main__':
    # 2. Create process object using process class
    # target: Specifies the name of the function executed by the process
    sing_process = multiprocessing.Process(target = sing)
    dance_process = multiprocessing.Process(target = dance)
    
    # 3. Use the process object to start the process and execute the specified task
    sing_process.start()
    dance_process.start()

4. The process executes tasks with parameters

4.1 the process executes tasks with parameters

Parameter name	explain
args	Pass parameters to the executed task in the way of Yuanzu
kwargs	Pass parameters to the executed task in the form of dictionary

4.2 use of args parameter

# target: the name of the function executed by the process
# args: it means to pass parameters to the function in the way of Yuanzu
sing_process = multiprocessing.Process(target = sing, args=(3,))
sing_process.start()

4.3 use of kwargs parameters

# target: the name of the function executed by the process
# kwargs: means to pass parameters to a function in the form of a dictionary
dance_process = multiprocessing.Process(target = dance, kwargs={"num":3})
dance_process.start()

The process executes a task with parameters py

# 1. Import process package
import multiprocessing
import time

# sing
def sing(num):
	for i in range(num)
		print("sing...")
		time.sleep(0.5)

# dance
def dance(num)
	for i in range(num)
		print("dance...")
		time.sleep(0.5)
        
if __name__ = '__main__':
    # 2. Create process object using process class
    # target: Specifies the name of the function executed by the process
    # args: use ancestor method to pass parameters to the specified task
    # kwargs: pass parameters to functions using dictionaries
    sing_process = multiprocessing.Process(target = sing, args=(3,))
    dance_process = multiprocessing.Process(target = dance, kwargs={'num':2})
    
    # 3. Use the process object to start the process and execute the specified task
    sing_process.start()
    dance_process.start()

When using Yuanzu parameters, the order of parameters must be consistent
When using dictionary to transfer parameters, you must ensure that the key and parameter name are consistent

5. Get process number

Role of process number:

When there are more and more processes in the program, if there is no way to distinguish the main process from the sub process, and there are different sub processes, then effective process management cannot be carried out. In order to facilitate management, in fact, each process has its own number

Two ways to get the process number

1. Get the current process number

os.getpid()

2. Get the current parent process number

os.getppid()

5.1 os. Use of getpid()

import os
pid = os.getpid()
print(pid)

5.2 os. Use of getppid()

def work():
	# Gets the number of the current process
    print("work Process number:", os.getpid())
    # Gets the number of the parent process
    print("work Parent process number:", os.getppid())

The process executes a task with parameters py

# 1. Import process package
import multiprocessing
import time
import os

# sing
def sing(num,name):
    print("Singing process pid: ",os.getpid())
    print("Parent process of the process pid: ",os.getppid())
	for i in range(num)
    	print(name)
		print("sing...")
		time.sleep(0.5)

# dance
def dance(num,name)
	print("Dance progress pid: ",os.getpid())
    print("Parent process of dance process pid: ",os.getppid())
	for i in range(num)
    	print(name)
        print("dance...")
        time.sleep(0.5)
        
if __name__ = '__main__':
    print("Of the main process pid: "，os.pid())
    # 1. Main process, create sub process object and specify the task name to be executed
    sing_process = multiprocessing.Process(target = sing, args=(3,"Xiao Ming"))
    dance_process = multiprocessing.Process(target = dance, kwargs={'num':2,'name':"Xiao Hong"})
    
    # 2. Start the subprocess and execute the task
    sing_process.start()
    dance_process.start()

6. Process considerations

6.1 the main process will wait until the execution of all sub processes is completed

import multiprocessing
import time

def work()
	for i in range(10):
        print("at work...")
        time.sleep(0.2)

if __name__ = '__main__':
    # Create child process
    work_process = multiprocessing.Process(target=work)
    work_process.start()
	# Let the main process wait for 1s
    time.sleep(1)
    print("The execution of the main process is complete!")

6.2 setting the main daemon

import multiprocessing
import time

def work()
	for i in range(10):
        print("at work...")
        time.sleep(0.2)

if __name__ = '__main__':
    # Create child process
    work_process = multiprocessing.Process(target=work)
    # Set the main process to be guarded. After the main process exits, the child process will be destroyed directly and the code of the child process will no longer be executed
    work_process.deamon = True
    work_process.start()
	# Let the main process wait for 1s
    time.sleep(1)
    print("The execution of the main process is complete!")

6.3 key points of knowledge

In order to ensure the normal operation of the child process, the main process will wait for all child processes to be destroyed after execution. The purpose of setting the guardian main process is that the main process exits and the child process is destroyed, so that the main process will not wait for the child process to execute.

Set the mode of main daemon: child process object daemon = True

7. Case - high concurrency copy device of intelligence teaching video folder (multi process)

requirement analysis

1. Whether the target folder exists. If it does not exist, it will be created. If it does not exist, it will not be created
2. Traverse all files in the source folder and copy to the destination folder
3. Multi task is realized by process to complete high concurrent copy

Implementation steps

1. Define the path where the source folder is located and the path where the destination folder is located

# 1. Define the source file directory and the destination folder directory
source_ dir = "python Teaching video"
dest_ dir = "/home/python/desktop/test"

2. Create destination folder

try:
    # 2. Create destination folder directory
    os.mkdir(dest_dir)
except:
    print("The destination folder already exists, Not created~")

3. Via OS Listdir gets the list of files in the source directory

# 3. List all the files in the original file
file_list = os.listdir(source_dir)
print(file_list)

4. Traverse each file and define a function to copy the file

# 4. The for loop copies each file in turn
for file_name in file_list:
    copy_work(file_name,source_dir,dest_dir)

5. Multi task is realized by process to complete high concurrent copy

for file_name in file_list:
    sub_process = multiprocessing.Process(target=copy_work,args=(file_name,source_dir,dest_dir))
    sub_process.start()

Implementation steps of document price copy function

def copy_work(file_name,source_dir,dest_dir)
	# 1. The path where the source and destination files are spliced
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    print(source_path,'------>',dest_path)
    # 2. Open the source file and create the directory file
    with open(source_path,"rb") as source_file:
        with open(dest_path,"wb") as dest_file:
            # 3. Read the contents of the source file and write to the target file (loop)
            while True:
                # Loop read data
                file_data = source_file.read(1024)
                if file_data:
                    # Write loop to target file
                    dest_file.write(file_data)
                else:
                    break

Multi - process to realize the copy of intelligence teaching video py

import multiprocessing
import os

def copy_work(file_name,source_dir,dest_dir)
	# 1. The path where the source and destination files are spliced
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    print(source_path,'------>',dest_path)
    # 2. Open the source file and create the directory file
    with open(source_path,"rb") as source_file:
        with open(dest_path,"wb") as dest_file:
            # 3. Read the contents of the source file and write to the target file (loop)
            while True:
                # Loop read data
                file_data = source_file.read(1024)  # 1024 bytes read at a time
                if file_data:
                    # Write loop to target file
                    dest_file.write(file_data)
                else:
                    break
    
if __name__ == '__main__'
    # 1. Define source folder and destination folder
    source_dir = "./python Teaching video"
    dest_dir = "./test"
    # 2. Create target folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The destination folder already exists, Not created~")
    # 3. Read the file list of the source folder
    file_list = os.listdir(source_dir)
    # 4. Traverse the file list to copy
    for file_name in file_list:
        # copy_work(file_name,source_dir,dest_dir)
        # 5. Use process to realize multi task and complete high concurrent copy
        sub_process = multiprocessing.Process(target=copy_work,args=(file_name,source_dir,dest_dir))
        sub_process.start()

Multithreading

The process is the smallest unit to allocate resources. Once a process is created, certain resources will be allocated, just as it is necessary to open two QQ software to chat with two people about QQ. It is a waste of resources.
Thread is the smallest unit of program execution. In fact, a process is only responsible for allocating resources, and threads use these resources to execute programs, that is, a process is a container of threads. At least one thread in a process is responsible for executing programs At the same time, the thread itself does not have system resources and only needs a little essential resources in operation, but it can share all the resources owned by the process with other threads belonging to the same process. This is like opening two windows (two threads) to chat with two people through a QQ software (one process), which not only realizes multitasking, but also saves resources.

1 Introduction to multithreading

1.3 role of multithreading

2 multithreading to complete multitasking

2.1 thread creation steps

1. Import thread module
import threading
2. Create thread objects through thread classes
Thread object = threading Thread (target = task name)
3. Start the thread to execute the task
Thread object start()

2.2 creating thread objects through thread classes

Thread object = threading Thread (target = task name)

Parameter name	explain
target	The name of the target task to be executed, which refers to the function name (method name)
name	Thread name, - generally not set
group	Thread group. Currently, only None can be used

2.3 code for thread creation and startup

import threading

# Create child thread
sing_thread = threading.Thread(target=sing)
# Create child thread
dance_thread = threading.Thread(target=dance)
# Start thread
sing_thread.start()
dance_thread.start()

Multitasking with multithreading

# 1. Import thread module
# 2. Create thread object
# 3. Start the thread and execute the task
import time
import threading
def sing():
    for i in range(3):
        print("sing...")
        time.sleep(1)
        
def dance():
    for i in range(3):
        print("dance")
        time.sleep(1)

if __name__ == '__main__':
    # sing()
    # dance()
    sing_thread = threading.Thread(target=sing)
    dance_thread = threading.Thread(target=dance)
    sing_thread.start()
    dance_thread.start()
    

3 threads execute tasks with parameters

3.1 threads execute tasks with parameters

Parameter name	explain
args	Pass parameters to the executed task in the way of Yuanzu
kwargs	Pass parameters to the executed task in the form of dictionary

3.2 use of args parameter

# target: the name of the function executed by the process
# args: it means to pass parameters to the function in the way of Yuanzu
sing_thread = threading.Thread(target = sing, args=(3,))
sing_thread.start()

3.3 use of kwargs parameters

# target: the name of the function executed by the process
# args: it means to pass parameters to the function in the way of Yuanzu
sing_thread = threading.Thread(target = sing, kwargs={"count":3})
sing_thread.start()

4 end order of main thread and sub thread

import time
import threading

def work():
    for i in range(10):
        print("work...")
        time.sleep(0.2)

if __name__ == '__main__':
    sub_thread = threading.Thread(target=work)
    sub_thread.start()
    # The main thread is delayed by 1 second
    time.sleep(1)
    print("over")

4.1 setting the main daemon thread

# To guard the main thread mode 1 Daemon = true daemon main thread
work_thread = threading.Thread(target=work,daemon=True)
# Set main thread guard mode 2
# work_thread.setDaemon(True)
work_thread.start()
# The main thread is delayed by 1 second
time.sleep(1)
print("over")

5 execution order between threads

Inter thread execution order

5.1 execution between threads is out of order

for i in range(5):
    sub_thread = threading.Thread(target=task)
    sub_thread.start()

5.2 get the current thread information

# Through current_ The thread method gets the thread object
current_thread = threading.current_thread()
# Through current_ The thread object can know the relevant information of the thread, such as the order in which it is created
print(current_thread)

Execution order between threads

import threading
import time

def task():
    time.sleep(1)
    # current_thread: get the thread object of the current thread
    thread = threading.current_thread()
    print(thread)

if __name__ = '__main__':
    for i in range(5):
        sub_thread = threading.Thread(target=task)
        sub_thread.start()
        
 # Conclusion: the execution between multithreads is disordered and scheduled by CPU

6 process and thread comparison

6.1 relationship comparison

1. Threads are attached to processes. Without processes, there will be no threads.
2. A process provides one thread by default, and a process can create multiple threads.

6.2 difference and comparison

1. The resource cost of creating a process is greater than that of creating a thread
2. Process is the basic unit of operating system resource allocation, and thread is the basic unit of CPU scheduling
3. Threads cannot execute independently and must exist in the process

6.3 comparison of advantages and disadvantages

1. Process advantages and disadvantages

   Advantages: multiple cores can be used

   Disadvantages: high resource overhead

2. Thread advantages and disadvantages

   Advantages: low resource overhead

   Disadvantages: multi core cannot be used

Case 7 - high concurrency copy device for educational video folder (multithreading)

import threading
import os

def copy_work(file_name,source_dir,dest_dir)
	# 1. The path where the source and destination files are spliced
    source_path = source_dir+"/"+file_name
    dest_path = dest_dir+"/"+file_name
    print(source_path,'------>',dest_path)
    # 2. Open the source file and create the directory file
    with open(source_path,"rb") as source_file:
        with open(dest_path,"wb") as dest_file:
            # 3. Read the contents of the source file and write to the target file (loop)
            while True:
                # Loop read data
                file_data = source_file.read(1024)  # 1024 bytes read at a time
                if file_data:
                    # Write loop to target file
                    dest_file.write(file_data)
                else:
                    break
    
if __name__ == '__main__'
    # 1. Define source folder and destination folder
    source_dir = "./python Teaching video"
    dest_dir = "./test"
    # 2. Create target folder
    try:
        os.mkdir(dest_dir)
    except:
        print("The destination folder already exists, Not created~")
    # 3. Read the file list of the source folder
    file_list = os.listdir(source_dir)
    # 4. Traverse the file list to copy
    for file_name in file_list:
        # copy_work(file_name,source_dir,dest_dir)
        # 5. Use process to realize multi task and complete high concurrent copy
        sub_thread = threading.Thread(target=copy_work,args=(file_name,source_dir,dest_dir))
        sub_thread.start()