Yesterday afternoon, my boss approached me with a stern expression: “Why is this data processing so slow? The clients are complaining.” I glanced at the screen and noticed that the progress bar was almost stuck in place, leaving me speechless. I believe most people have had similar experiences; even when the code logic is flawless, large files can cause JSON to freeze up completely. At that moment, my boss was there, expressing his frustration. Later, I discovered an open-source project on GitHub called SimdJSON, which has already garnered 21.6k stars and perfectly solved my problem. Many of you might feel that traditional JSON parsers are like trying to drink soup with chopsticks; they work, but the efficiency is just too low. Let me give you a few examples. RapidJSON? Although it claims to be fast, it still struggles with large files; the standard parser? I feel like it crawls at a snail’s pace, with CPU usage being quite outrageous; as for those third-party libraries, they either lack functionality or have memory leaks. After going through the options, there weren’t many that were satisfactory! SimdJSON is different; it is 4 times faster than RapidJSON and 25 times faster than JSON for Modern C++, capable of processing several GBs of JSON data per second! What does that mean? Let me put it this way: a file that used to take you 1 minute to process now only takes 2.4 seconds!
The reason SimdJSON can achieve such speed is that it utilizes the SIMD instruction set. To put it simply, a regular parser works in a single-threaded manner, like a worker assembling parts one by one, while SimdJSON allows workers to handle multiple parts simultaneously. Do you understand? The highlight of SimdJSON is not just its speed; it also has a comprehensive set of features. It includes strict JSON validation, ensuring that format issues do not lead to sudden crashes; it supports UTF-8, making Chinese processing a breeze; and it also supports NDJSON, which is a boon for streaming data processing.
Although the core of SimdJSON is written in C++, it supports bindings for multiple languages, covering most mainstream languages. So no matter what technology stack you are using, you can benefit from the performance improvements that SimdJSON offers. As the volume of data we handle continues to grow and user demands for response speed increase, if you are facing similar frustrations, you might want to give this open-source project a try. Here’s the link: https://github.com/simdjson/simdjson