Infi-chu:

http://www.cnblogs.com/Infi-chu/

1, Simple cyclic serial
Cycle by cycle, which takes the longest time, is the synthesis of all the time

import requests
url_list = [
    'http://www.baidu.com',
    'http://www.pythonsite.com',
    'http://www.cnblogs.com/'
]

for url in url_list:
    result = requests.get(url)
    print(result.text)

2, Through thread pool
The overall time consumption is the longest of all connections, which is much faster than the cycle

import requests
from concurrent.futures import ThreadPoolExecutor

def fetch_request(url):
    result = requests.get(url)
    print(result.text)

url_list = [
    'http://www.baidu.com',
    'http://www.bing.com',
    'http://www.cnblogs.com/'
]
pool = ThreadPoolExecutor(10)

for url in url_list:
    #Get a thread from the thread pool, and the thread executes the fetch request method
    pool.submit(fetch_request,url)

pool.shutdown(True)

3, Thread pool + callback function
Defined a callback function

from concurrent.futures import ThreadPoolExecutor
import requests


def fetch_async(url):
    response = requests.get(url)

    return response


def callback(future):
    print(future.result().text)


url_list = [
    'http://www.baidu.com',
    'http://www.bing.com',
    'http://www.cnblogs.com/'
]

pool = ThreadPoolExecutor(5)

for url in url_list:
    v = pool.submit(fetch_async,url)
    #Call the callback function here
    v.add_done_callback(callback)

pool.shutdown()

4, Through process pool
The way to access the process pool also depends on the time-consuming one. However, compared with the thread, the process needs more resources. At the same time, this is the IO operation when accessing the url, so the thread pool is better than the process pool

import requests
from concurrent.futures import ProcessPoolExecutor

def fetch_request(url):
    result = requests.get(url)
    print(result.text)

url_list = [
    'http://www.baidu.com',
    'http://www.bing.com',
    'http://www.cnblogs.com/'
]
pool = ProcessPoolExecutor(10)

for url in url_list:
    #Get a thread from the process pool, and execute the fetch request method in the subprogram
    pool.submit(fetch_request,url)

pool.shutdown(True)

5, Process pool + callback function
The effect of this method is the same as that of thread + callback function. Relatively speaking, starting a process wastes more resources than starting a thread

from concurrent.futures import ProcessPoolExecutor
import requests


def fetch_async(url):
    response = requests.get(url)

    return response


def callback(future):
    print(future.result().text)


url_list = [
    'http://www.baidu.com',
    'http://www.bing.com',
    'http://www.cnblogs.com/'
]

pool = ProcessPoolExecutor(5)

for url in url_list:
    v = pool.submit(fetch_async, url)
    # Call the callback function here
    v.add_done_callback(callback)

pool.shutdown()