1. Why Do We Need Dual Protocol Support?
Modern websites are like hybrid cars, featuring both standard pages (HTTP) and encrypted content (HTTPS). For instance, when scraping e-commerce websites, the product detail page might be HTTP, but the user comments interface requires HTTPS. Using a single protocol proxy can lead to two troublesome issues: either encrypted requests fail, or standard requests get intercepted.
Here, we recommend using ipipgo dynamic residential IP, which has intelligent protocol recognition that can automatically switch to adapt. For example: when the target website is detected to have HTTPS enabled, the system will automatically allocate an IP that supports SSL handshake, achieving a success rate over 40% higher than ordinary proxies.
2. Setup Process for Beginners
Step 1: Prepare the Ingredients 1. Rent a cloud server (starting with 2 cores and 4GB, recommended Ubuntu system) 2. Open the ipipgo console and create a “Dual Protocol Dedicated IP Pool” (remember to check the “automatic protocol adaptation” option) 3. Install Nginx on the server, version recommended is above 1.18
Step 2: Configure the Secret Sauce Modify the Nginx configuration file (/etc/nginx/nginx.conf):
http { server { listen 8888;
# HTTP processing channel
location /http {
proxy_pass http://$target_url;
proxy_set_header X-Real-IP ${ipipgo dynamic IP};
}
# HTTPS processing channel
location /https {
proxy_ssl_server_name on;
proxy_pass https://$target_url:443;
proxy_set_header Host $host;
}
}
After saving, run sudo nginx -s reload to take effect. The beauty of this configuration lies in differentiating protocols using different paths, which is more resource-efficient than opening multiple ports.
Step 3: Bind IP Resources Go to the ipipgo developer center to obtain the API key and set two key rules: 1. When HTTPS requests fail 3 times, automatically switch to an IP that supports TLS 1.3 2. Automatically clean up IPs used for over 12 hours every day at midnight
3. Performance Optimization Trio
| Problem Manifestation | Solution | ipipgo Function |
|---|---|---|
| Slow access speed | Enable HTTP/2 protocol support | Dynamic IPs prioritize assigning HTTP/2 nodes |
| Frequent CAPTCHA appearances | Limit each IP to ≤60 requests per hour | Traffic monitoring auto-fuse |
| HTTPS certificate errors | Update server CA certificate package | Provide pre-signed certificate downloads |
4. Enterprise-Level Solution Expansion
If you need to handle over 100,000 requests daily, it is recommended to adopt a distributed architecture: 1. Deploy one server each in Tokyo, Singapore, and Frankfurt 2. Use ipipgo’s intelligent routing API, allowing US users to automatically connect to Silicon Valley nodes, and Japanese users to connect to Tokyo nodes 3. Use Redis to cache high-frequency requests, reducing duplicate scraping A financial company used this method, tripling their data collection speed.
5. Common Issues Troubleshooting Guide
Q: What are the advantages of building a proxy compared to directly using ipipgo’s API?A: Building your own has three major advantages: 1. Customizable caching strategies (e.g., image cache for 1 hour, text cache for 5 minutes) 2. Can perform in-depth analysis combined with internal monitoring systems 3. Lower long-term costs (cost of 100,000 requests drops by 60%)
Q: Why do HTTPS requests sometimes get interrupted?A: Check three places: 1. Whether proxy_ssl_server_name is enabled in the Nginx configuration 2. Whether the server time is accurate (timezone errors can lead to SSL handshake failures) 3. Whether ipipgo’s IP pool has enabled “high encryption level filtering”
Q: How to test if the proxy is working normally?A: Use this combination to test: 1. Visit http://httpbin.org/ip to see if the returned IP changes 2. Visit https://www.cloudflare.com/cdn-cgi/trace to check if HTTPS is accessible 3. Check in the ipipgo console for the IP usage report to confirm the switching frequency is normal
Q: Is server maintenance required?A: Just do these three things every month: 1. Update Nginx security patches 2. Clean up log files (to prevent hard disk overflow) 3. Refresh IP binding relations in the ipipgo backend
By following this guide, you can set up a stable and usable dual protocol proxy in as little as 1 hour. Now, use ipipgo’s free trial package to experience the advantages of 90 million residential IP resources, with all premium features available during the testing period, especially suitable for enterprises needing large-scale data collection.
