One Line of C++ Code, 100x Performance Improvement? My 30-Round Battle with a ‘Hell-Level’ Bug

Note: This article and title were primarily written by Gemini, so you may notice a difference from my previous writing style.

Recently, I was working on a deduplication task for a large-scale text dataset. Deduplication is necessary before training large models, as it not only reduces computational load (there is an enormous amount of duplicate data on the internet) but also improves model performance (refer to this ACL paper: Lee, Katherine, et al. “Deduplicating training data makes language models better.” arXiv preprint arXiv:2107.06499 (2021)). It can even prevent data leakage. If you have dealt with big data, you know that when the data volume reaches tens of millions or even hundreds of millions, a seemingly simple “deduplication” operation can turn into a performance disaster. The pure Python implementation was slower than a turtle, so I decided to take matters into my own hands and write a high-performance deduplication core in C++.

I set off with confidence, thinking that with C++, multithreading, and Faiss as my “silver bullets,” I could easily achieve a hundredfold performance improvement.

However, I was wrong.

I spent a day embroiled in a “hell-level” battle with bugs. This article is a retrospective of that battle, documenting how I transformed a buggy, poorly performing C++ prototype into an industrial-grade high-performance deduplication tool. I hope my experience helps you avoid every pitfall I encountered.

My Arsenal: A Three-Stage Deduplication Pipeline

My goal was clear: not only to remove completely identical texts but also to identify those “similar-looking” paraphrases and paragraphs that were slightly modified after copy-pasting. To achieve this, I designed a classic three-stage deduplication pipeline:

Stage One: Content-Defined Chunking (CDC)

• Goal: Quickly eliminate massive amounts of precisely duplicate text blocks.
• Technique: Use a rolling hash on the text byte stream to slide over and define chunk boundaries based on certain characteristics of the hash value (e.g., hash % N == 0). The magic of this method lies in the fact that regardless of where a piece of content appears, its chunking method is likely to remain consistent.
• Note: The original ACL paper used a sorted suffix array, which has some efficient O(n) algorithms, but for large amounts of data, the efficiency is still insufficient. I implemented this algorithm but found that CDC could achieve about 90% of the deduplication effect of the suffix array while being ten times faster, so I ultimately defaulted to using CDC. You can, of course, configure the command line to use the suffix array.
• Stage Two: Document Fingerprinting (SimHash)

• Goal: Generate a compact “fingerprint” for documents that have undergone the first round of cleaning. Documents with similar content will have fingerprints that are also very close.
• Technique: The SimHash algorithm, which can compress a long document into a 64-bit or 128-bit binary number. The Hamming distance between fingerprints (the number of differing binary bits) directly reflects the similarity of the original texts.
• Stage Three: Nearest Neighbor Search (Faiss)

• Goal: Efficiently find all “neighbor pairs” with a Hamming distance below a certain threshold among massive document fingerprints.
• Technique: The Faiss library from Facebook AI, a tool designed for dense/binary vector similarity search.

This architecture is theoretically flawless, and performance should be explosive.

I quickly implemented the first version using C++, OpenMP (multithreading), and pybind11 (Python bindings).

What was the result? It was only 10 times faster than Python, and the deduplication rate was still questionable.

The battle began.

First Battle: Ghost Locks and the “Illusion” of Parallelization

Symptoms: CPU cores fully utilized, but the program’s “wall clock time” decreased disproportionately.

Diagnosis: I confidently used #pragma omp parallel for, thinking I had activated “rage mode.” But a tiny detail in the code revealed the truth:

// Pseudocode
#pragma omp parallel for

for (doc in all_docs) {
    chunks = get_chunks(doc);
    for (chunk in chunks) {
    lock(global_seen_chunks_mutex); // Global lock
    if chunk not in global_seen_chunks:
        // ...
    unlock(global_seen_chunks_mutex);
}

All threads were frantically contending for the same global lock! My parallelization turned into a joke, with most threads waiting in line, and the CPU idling.

Lesson #1: The number one enemy of parallelization is lock contention.

Solution: Refactor! Adopt a “parallel computation, serial merging” or “thread-local storage” model. I split the logic into three steps:

1. Parallel Chunking: Each thread processes its own document, storing the (hash, chunk) results in its own local list. Completely lock-free.
2. Serially Build Global Set: After the parallel part ends, use a fast single-threaded loop to traverse all thread results and build a global seen_hashes set.
3. Parallel Reconstruction of Text: Restart the parallel loop, with each thread independently reconstructing its assigned document based on the global set. Completely lock-free.

This change doubled the performance again. But I still had a nagging feeling that the deduplication rate was still off.

Second Battle: The “Butterfly Effect” of SimHash and the Betrayal of AVX2

Symptoms: The deduplication rate was surprisingly low and did not meet expectations. Similar documents “slipped by” unnoticed.

Diagnosis: I began to suspect the implementation of SimHash. In pursuit of extreme performance, I used the AVX2 SIMD instruction set to accelerate calculations. The code looked cool:

// Pseudocode - Incorrect AVX2 Implementation

__m256i delta = _mm256_blendv_epi8(weights_sub, weights_add, prng_hash);

_mm256_blendv_epi8 selects by byte (8-bit). However, my intention was to select the entire 32 bits based on the sign bit of a 32-bit integer. As a result, a delta value was “torn apart” and “stitched together” by four different bytes, resulting in a meaningless random number. Naturally, the generated SimHash fingerprints were a mess.

Lesson #2: The devil is in the details, especially in SIMD programming. A tiny difference in one instruction can lead to a complete failure of the algorithm logic.

Solution: Use the correct instruction _mm256_srai_epi32 to first generate a perfect 32-bit mask, then perform blendv.

// Correct Implementation

__m256i mask = _mm256_srai_epi32(prng_hash, 31);

__m256i delta = _mm256_blendv_epi8(weights_sub, weights_add, mask);

The bug was fixed, and SimHash could finally correctly identify similar documents. But a bigger problem surfaced.

Third Battle: The “Backstab” of string_view — The Hell of Dangling Pointers

Symptoms: The deduplication rate was still lower than the initial slow version, and there were occasional strange UnicodeDecodeError occurrences.

Diagnosis: This is a classic trap in C++. In pursuit of zero-copy, I used std::string_view in many function interfaces.

// Pseudocode - Fatal use of string_view

std::vector<string_view> get_chunks(const std::string& doc) {
    // ...
    // text.substr() creates a temporary std::string
    // string_view points to the memory of this temporary object
    views.push_back(doc.substr(...));
    return views;
    // When the function returns, the temporary string is destroyed, and all string_view become dangling pointers!
}

Subsequent code accessing these dangling pointers read garbage memory that had already been released and overwritten. Naturally, these garbage bytes could not be UTF-8 decoded when returned to Python.

Of course, I eventually clarified my thoughts and implemented a safe version of string_view.

Lesson #3: string_view is a promise, not a container. It does not own the data. You must ensure 100% that the data it points to has a longer lifetime than itself. Maintaining this guarantee is extremely difficult in complex parallel and data flow scenarios.

Solution: Abandon the fantasy and return to basics. In places where ownership is unclear, honestly use std::string. I changed all functions that produce new chunks to return std::string and used std::move during data transfer to ensure efficient ownership transfer.

Fourth Battle: The Final Boss — UTF-8 Characters Shredded by CDC

Symptoms: After fixing all the above bugs, UnicodeDecodeError stubbornly persisted.

Diagnosis: At this point, I had ruled out all software engineering bugs. The problem must lie in the essential interaction between the algorithm and the data.

The Truth: CDC is a byte stream algorithm. It does not recognize Unicode at all! When it encounters a Chinese character composed of three bytes (for example, 你, 0xE4 0xBD 0xA0), its rolling hash could very well decide to split in the middle of these three bytes!

The result was that I created a bunch of text blocks that began and ended with “half a Chinese character.” These corrupted byte sequences were stitched together, causing the decoder to crash at the last moment when returning to Python.

Lesson #4: When dealing with Unicode text, never assume that byte stream operations are safe.

Solution: Before the data leaves C++ and returns to Python, perform a “sanitization” step. I implemented a clean_utf8 function that traverses the final text and removes all byte sequences that do not conform to UTF-8 encoding rules.

// Pseudocode
for (text in final_texts_to_return) {
    cleaned_text = clean_utf8(text);
    // ...
}

With this final line of defense, all UnicodeDecodeError issues finally vanished.

Final Outcome: A “Battle-Tested” Deduplication Tool

After dozens of rounds of fierce battles, my initial simple C++ script has evolved into a completely different species:

• A robust C++ core library: Integrating OpenMP, Faiss, Abseil, AVX2, xxHash.
• A comprehensive Python wrapper (dedup_toolkit.py): Providing command-line deduplication tools.
• A clear API: Exposing interfaces through pybind11, specifically optimized for the usage flow of the Hugging Face datasets library.

How is the performance?

After fixing all bugs and allowing the CDC stage to truly take effect, the overall performance improvement far exceeded the initial 10 times. Depending on data redundancy, it easily achieved 50-100 times acceleration. A Python script that used to take a whole day can now be completed in less than half an hour. For the 1.3GB cc news data, deduplication can be completed in 100 seconds, removing about 30% of the data. Experiments have shown that it does not affect the performance of the large model trained subsequently, and even slightly improves perplexity and accuracy.

The Biggest Lesson I Learned

Because my skills were relatively lacking, and for debugging convenience, my first version was actually based on a single-threaded Python implementation, which I then gradually optimized for performance. I later implemented it in C++, switched to a parallel implementation, and finally added various optimizations like zero-copy, SIMD, and lock-free. Fortunately, I approached it step by step, so I started with a simple but correct version. With this correct version, I could easily determine whether subsequent versions had bugs, which allowed me to complete development in just one day. So, start from simple.

In Conclusion

This experience deeply taught me that high-performance programming is not just about piling up algorithms and clever tricks. It is a system engineering process that constantly weighs correctness, performance, and maintainability.

• The enemy of parallelization is locks.
• The enemy of zero-copy is lifecycle.
• The enemy of byte stream algorithms is Unicode.
• Start from simple: Always write a simple but easily correct solution first, then optimize significant bottlenecks, and profile later.

I hope my “bloody history” can help you avoid some detours on your future performance optimization journey. If you are also interested in large-scale data processing, feel free to discuss in the comments!

Appendix: My Project Overview

https://github.com/conanhujinming/text_dedup

This tool is now a fully functional high-performance text deduplication library that you can call using simple Python scripts.

Core Features:

• deduplicate_cpp(): A one-stop function that takes a list of documents and outputs a cleaned and deduplicated list of documents (duplicates are marked as None), specifically designed for datasets.map.

Usage Example:

import dedup_cpp_core

from datasets import Dataset
# 1. Prepare data

docs = ["doc a", "doc a is a copy", "doc b"]
dataset = Dataset.from_dict({'text': docs})
# 2. Call core
results = dedup_cpp_core.deduplicate_cpp(dataset['text'], ...)
# 3. Apply results
dataset = dataset.add_column("updated", results)
final_ds = dataset.filter(lambda x: x['updated'] is not None)

Thank you for reading!