preface
Among Python's many HTTP clients, the most famous are requests, aiohttp and httpx.
Requests can only send synchronization requests without the help of other third-party libraries; AIO HTTP can only send asynchronous requests; httpx can send both synchronous and asynchronous requests.
So how to choose
- Only synchronous requests are sent, but they can cooperate with multi-threaded mutation steps.
- aiohttp is only used for asynchronous requests, but it can be synchronized with await.
- httpx can send synchronous or asynchronous requests, but the request speed is slightly worse than requests and asynchronous than AIO http
It is not recommended to use multithreading to make asynchronous requests. It is recommended to use asynchronous IO.
httpx features:
- Powerful, both synchronous and asynchronous.
- The syntax of synchronization requests and requests is basically the same, which is convenient for code migration.
- Although the performance is poor, it is not much worse and can be ignored.
Synchronization request
GET request
import httpx
r = httpx.get(
'https://www.psvmc.cn/login.json',
params={'keyword': '123'}
)
print("r.text:", r.text)
print("r.json():", r.json())
print("r.content:", r.content)
# Response code
print("r.status_code:", r.status_code)
# Coding of response
print("r.encoding:", r.encoding)
# Requested URL
print("r.url:", r.url)
# Cookie
print("r.cookies:", r.cookies)
# Response header
print("r.headers:", r.headers)
print("r.headers['Content-Type']:", r.headers['Content-Type'])
# Requested header
print("r.request.headers:", r.request.headers)result
r.text: {"code":0,"msg":"success","obj":{"name":"Xiao Ming","sex":"male","token":"psvmc"}}
r.json(): {'code': 0, 'msg': 'success', 'obj': {'name': 'Xiao Ming', 'sex': 'male', 'token': 'psvmc'}}
r.content: b'{"code":0,"msg":"success","obj":{"name":"\xe5\xb0\x8f\xe6\x98\x8e","sex":"\xe7\x94\xb7","token":"psvmc"}}'
r.status_code: 200
r.encoding: utf_8
r.url: https://www.psvmc.cn/login.json?keyword=123
r.cookies: <Cookies[]>
r.headers: Headers({'server': 'nginx/1.14.0 (Ubuntu)', 'date': 'Fri, 26 Nov 2021 02:23:03 GMT', 'content-type': 'application/json', 'content-length': '78', 'last-modified': 'Thu, 25 Nov 2021 10:57:01 GMT', 'connection': 'keep-alive', 'etag': '"619f6bfd-4e"', 'accept-ranges': 'bytes'})
r.headers['Content-Type']: application/json
r.request.headers: Headers({'host': 'www.psvmc.cn', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.21.1'})Judge return status code
r.status_code == httpx.codes.OK
POST request
Basic request
r = httpx.post('https://www.psvmc.cn/login.json', data={'key':'value'})
r = httpx.post('https://www.psvmc.cn/login.json', json={'key':'value'})File upload
files ={'upload-file': open('report.xls','rb')}
r = httpx.post(url, files=files)Documents and data
data = {'message': 'Hello, world!'}
files = {'file': open('report.xls', 'rb')}
r = httpx.post("https://httpbin.org/post", data=data, files=files)binary data
content = b'Hello World'
response = httpx.post('http://127.0.0.1:5000/test/post', content=content)Other requests
r = httpx.put('https://www.psvmc.cn/login.json', data={'key':'value'})
r = httpx.delete('https://www.psvmc.cn/login.json')
r = httpx.head('https://www.psvmc.cn/login.json')
r = httpx.options('https://www.psvmc.cn/login.json')Set timeout
import httpx
r = httpx.get('https://www.psvmc.cn/login.json', timeout=1)
print(r.text)SSL
response = httpx.get('https://example.org', verify='../../client.pem')Alternatively, you can set verify to False to disable SSL authentication:
response = httpx.get('https://example.org', verify=False)Custom Header
headers ={'user-agent':'psvmc/0.0.1'}
r = httpx.get(url, headers=headers)Authentication mode
HTTP x supports basic and digest HTTP authentication.
To provide basic authentication credentials, pass a plain text str or bytes object of 2 tuples to the request function as an auth parameter:
import httpx
r = httpx.get(
"https://www.psvmc.cn/login.json",
auth=("my_user", "password123")
)
print(r.text)To provide credentials for digest authentication, you need DigestAuth to instantiate an object with plain text username and password as parameters. The object can then be passed to the above request method as an auth parameter:
import httpx
auth = httpx.DigestAuth("my_user", "password123")
r = httpx.get("https://www.psvmc.cn/login.json", auth=auth)
print(r.text)Asynchronous request
import httpx
import asyncio
async def myrequest():
async with httpx.AsyncClient() as client:
resp = await client.get(
'https://www.psvmc.cn/login.json',
params={'keyword': '123'}
)
result = resp.text
print(result)
loop = asyncio.get_event_loop()
loop.run_until_complete(myrequest())response
Common responses
print("r.text:", r.text)
print("r.json():", r.json())
print("r.content:", r.content)
# Response code
print("r.status_code:", r.status_code)
# Coding of response
print("r.encoding:", r.encoding)
# Requested URL
print("r.url:", r.url)
# Cookie
print("r.cookies:", r.cookies)
# Response header
print("r.headers:", r.headers)
print("r.headers['Content-Type']:", r.headers['Content-Type'])
# Requested header
print("r.request.headers:", r.request.headers)Flow response
For large downloads, you may need to use a streaming response that does not immediately load the entire response body into memory.
You can stream the binary content of the response
import httpx
with httpx.stream("GET", "https://www.psvmc.cn/login.json") as r:
for data in r.iter_bytes():
print(data)Or response text
import httpx
with httpx.stream("GET", "https://www.psvmc.cn/login.json") as r:
for text in r.iter_text():
print(text)Or progressive streaming text
import httpx
with httpx.stream("GET", "https://www.psvmc.cn/login.json") as r:
for line in r.iter_lines():
print(line)HTTPX will use common line endings to standardize all cases to \ n.
In some cases, you may want to access the original bytes on the response without applying any HTTP content decoding. In this case, any content encoded by the web server, such as gzip, deflate or brotli, will not be automatically decoded.
import httpx
with httpx.stream("GET", "https://www.psvmc.cn/login.json") as r:
for chunk in r.iter_raw():
print(chunk)If you use streaming response in any of the above ways, the response.contentand response.text property will not be available and an error will be raised if accessed. However, you can also use the response flow function to conditionally load the response body:
import httpx
with httpx.stream("GET", "https://www.psvmc.cn/login.json") as r:
if int(r.headers['Content-Length']) < 1000:
r.read()
print(r.text)Binary loading as picture
from PIL import Image from io import BytesIO i =Image.open(BytesIO(r.content))