anti-scraping

Reverse Engineering of the msToken Parameter: A Complete Analysis of the JS Encryption Algorithm

by
Reverse Engineering of the msToken Parameter: A Complete Analysis of the JS Encryption Algorithm

Brothers, last time we detailed the reverse engineering process of a certain headline’s a_bogus parameter, today we will tackle another key parameter—msToken. This parameter also plays an important role in the /api/pc/list/feed interface, dynamically changing with each request, and is a crucial part of the anti-scraping mechanism. 🧩 Core Analysis Approach Through packet analysis, we … Read more

Fearless Against Anti-Scraping? This Open Source HTTP Client Stands Strong!

by
Fearless Against Anti-Scraping? This Open Source HTTP Client Stands Strong!

Hello everyone, I am Octopus Cat. Those who have worked with web scraping know that the HTTP request step is often the “key to success or failure”. If you are just requesting some public APIs, Go’s net/http standard library is completely sufficient. However, once you encounter websites with anti-scraping detection, the situation becomes complicated: Some … Read more

Advanced Python Web Scraping: How to Bypass Anti-Scraping Mechanisms to Retrieve Any Data You Want?

by
Advanced Python Web Scraping: How to Bypass Anti-Scraping Mechanisms to Retrieve Any Data You Want?

In modern web scraping, due to the complexity of anti-scraping mechanisms, many developers often encounter obstacles when scraping data. Common anti-scraping techniques include IP restrictions, CAPTCHAs, User-Agent recognition, and JS rendering. To effectively cope with these anti-scraping mechanisms, we need to adopt corresponding strategies. 1. Set a Reasonable User-Agent: Disguise as a Browser Some websites … Read more