Hello everyone, today I want to introduce a particularly powerful Python library – mitmproxy! It is an intercepting proxy tool that supports HTTP/HTTPS, allowing us to easily monitor and modify network requests. Whether for web scraping or API testing, it can save us a lot of trouble. I have been using it for several years, and today I will share my experience!
Installation and Configuration
We need to install mitmproxy:
pip install mitmproxy
mitmproxy provides three usage modes:
- mitmproxy: Terminal interactive interface
- mitmweb: Web interactive interface
- mitmdump: Command line interface
Today, I will mainly introduce how to call mitmdump using Python code.
Writing a Simple Interceptor
Let’s see how to write a basic HTTP request interceptor:
from mitmproxy import ctx
def request(flow):
# Print the request URL
ctx.log.info(f"Request URL: {flow.request.pretty_url}")
# Modify request headers
flow.request.headers["User-Agent"] = "My Custom User Agent"
def response(flow):
# Modify response content
if "text/html" in flow.response.headers["content-type"]:
html = flow.response.text
html = html.replace("Original Text", "Replaced Text")
flow.response.text = html
Tip: Save the above code as script.py, then run:<span>mitmdump -s script.py</span> to start the proxy server!
Practical Case: API Testing Assistant
Let’s write a practical API testing tool:
from mitmproxy import ctx
import json
class APITester:
def __init__(self):
self.num_requests = 0
def request(self, flow):
self.num_requests += 1
# Log API request details
if "/api/" in flow.request.pretty_url:
ctx.log.info(f"Found API request #{self.num_requests}")
ctx.log.info(f"Request Method: {flow.request.method}")
ctx.log.info(f"Request URL: {flow.request.pretty_url}")
# Print request body
if flow.request.content:
try:
body = json.loads(flow.request.content)
ctx.log.info(f"Request Body: {json.dumps(body, indent=2)}")
except:
ctx.log.warn("Request body is not in JSON format")
def response(self, flow):
if "/api/" in flow.request.pretty_url:
# Check response status
ctx.log.info(f"Response Status Code: {flow.response.status_code}")
# Log response time
ctx.log.info(f"Response Time: {flow.response.timestamp_end - flow.request.timestamp_start:.2f} seconds")
addons = [APITester()]
Notes:
- Using mitmproxy to intercept HTTPS requests requires installing certificates
- When running the proxy server, ensure the port number (default 8080) is not occupied
- It is recommended to use it in a testing environment, do not modify requests casually in a production environment
Advanced Techniques
- Traffic Filtering: Focus only on specific requests
def request(flow):
# Only process requests for specific domains
if flow.request.pretty_url.startswith("https://api.example.com"):
ctx.log.info("Target request found!")
- Modifying Response Data:
def response(flow):
# Modify JSON response
if "application/json" in flow.response.headers["content-type"]:
data = json.loads(flow.response.content)
data["extra_field"] = "Test Data"
flow.response.content = json.dumps(data).encode()
Exercises:
- Try writing an interceptor that counts the number and size of all image requests
- How to implement a simple reverse proxy using mitmproxy?
Useful Tips
- During development, you can use the mitmweb interface to view request details, which is more intuitive
- For large projects, it is recommended to separate different functional interceptors into multiple classes
- You can use the logging module to save request logs to a file
- Be sure to handle exceptions to prevent script errors from affecting normal requests
Friends, today’s Python learning journey ends here! Remember to code along, and feel free to ask me any questions in the comments. mitmproxy is truly a powerful tool in Python development, mastering it will significantly enhance your development efficiency! Wishing everyone happy learning and continuous improvement in Python!
Like and Share
Let Money and Love Flow to You