Python httpx详解：新一代HTTP客户端的极致体验

Dxy1239310216

1156人浏览 · 2025-07-04 14:27:20

Dxy1239310216 · 2025-07-04 14:27:20 发布

引言

在Python的HTTP客户端库中，requests曾长期占据主导地位。然而，随着现代Web开发对性能、异步支持和协议兼容性的更高要求，httpx作为新一代HTTP客户端库应运而生。它由Django REST framework的作者Tom Christie开发，不仅兼容requests的API，还提供了异步支持、HTTP/2、连接池等先进特性。本文将全面解析httpx的核心功能、使用场景及实战技巧，帮助开发者高效驾驭现代网络请求。

一、httpx核心特性解析

1. 同步与异步双模式支持

httpx同时提供同步和异步API，满足不同场景需求：

同步模式：与requests高度兼容，适合简单请求。
异步模式：基于asyncio，支持高并发场景，通过AsyncClient实现。

示例：异步请求

import httpx
import asyncio

async def fetch_data():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://httpbin.org/get")
        print(response.json())

asyncio.run(fetch_data())

2. HTTP/1.1与HTTP/2无缝兼容

httpx默认支持HTTP/1.1，并可通过配置启用HTTP/2：

client = httpx.Client(http2=True)
response = client.get("https://httpbin.org/get")
print(response.http_version)  # 输出 "HTTP/2"

3. 丰富的功能集

客户端会话管理：通过Client对象复用配置（如headers、超时）。
重试机制：自动重试失败请求，提升可靠性。
认证支持：基本认证、摘要认证、令牌认证等。
流式响应：处理大文件或实时数据流。
代理支持：配置HTTP/HTTPS代理。

二、基础用法：从同步到异步

1. 安装与验证

pip install httpx  # 基础安装
pip install httpx[http2]  # 包含HTTP/2支持

验证安装：

import httpx
print(httpx.__version__)  # 输出版本号

2. 同步请求示例

GET请求

response = httpx.get("https://httpbin.org/get")
print(response.status_code)  # 200
print(response.json())  # 解析JSON响应

POST请求

response = httpx.post(
    "https://httpbin.org/post",
    data={"key": "value"},
    headers={"X-Custom-Header": "value"}
)
print(response.text)

3. 异步请求进阶

并发请求多个URL

async def fetch_multiple():
    async with httpx.AsyncClient() as client:
        tasks = [
            client.get("https://httpbin.org/get"),
            client.get("https://httpbin.org/ip")
        ]
        responses = await asyncio.gather(*tasks)
        for resp in responses:
            print(resp.json())

asyncio.run(fetch_multiple())

三、高级特性实战

1. 客户端会话管理

通过Client对象复用配置，提升性能：

with httpx.Client(headers={"User-Agent": "MyApp"}, timeout=10) as client:
    response = client.get("https://httpbin.org/get")
    print(response.json())

2. 重试机制配置

自动重试失败请求：

from httpx import Retry, HTTPTransport

transport = HTTPTransport(retries=3, retry_strategy=Retry(total=3))
client = httpx.Client(transport=transport)

3. 认证与安全

基本认证

response = httpx.get(
    "https://httpbin.org/basic-auth/user/pass",
    auth=("user", "pass")
)
print(response.status_code)  # 200表示认证成功

令牌认证

response = httpx.get(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

4. 流式响应处理

处理大文件或实时数据流：

with httpx.stream("GET", "https://httpbin.org/stream/10") as response:
    for chunk in response.iter_bytes():
        print(chunk)

四、httpx vs requests：如何选择？

特性	httpx	requests
异步支持	原生支持（AsyncClient）	不支持
协议版本	HTTP/1.1 + HTTP/2	HTTP/1.1
性能	异步模式下性能优异	同步模式性能一般
API设计	兼容requests，现代且灵活	简单易用，功能基础
依赖	依赖httpcore、h11等库	依赖少，安装简单

选择建议：

高并发场景：优先使用httpx的异步模式。
简单请求：requests更简洁，但httpx同步模式也可替代。
现代协议需求：HTTP/2支持选择httpx。

五、实战案例：构建爬虫与API调用

案例1：异步爬取多个网页

import httpx
import asyncio

async def scrape_urls(urls):
    async with httpx.AsyncClient() as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)
        return [resp.text for resp in responses]

urls = [
    "https://httpbin.org/get",
    "https://httpbin.org/ip",
    "https://httpbin.org/headers"
]

results = asyncio.run(scrape_urls(urls))
for idx, html in enumerate(results):
    print(f"Page {idx} content length: {len(html)}")

案例2：调用第三方API（带重试）

from httpx import Retry, HTTPTransport

transport = HTTPTransport(retries=3)
client = httpx.Client(transport=transport)

def fetch_api_data():
    response = client.get("https://api.example.com/data")
    response.raise_for_status()  # 抛出HTTP错误
    return response.json()

try:
    data = fetch_api_data()
    print(data)
except httpx.HTTPStatusError as e:
    print(f"请求失败: {e}")