1. Overview
Blosc (pronounced /blɒsk/) is a high-performance C library focused on binary data compression, providing an extremely simple C++ interface. Its design philosophy is fundamentally different from traditional compression libraries (such as zlib or LZMA): its primary goal is not the ultimate compression ratio, but the ultimate compression/decompression speed, aiming to achieve “the speed of compressing data that is even faster than directly transferring data from memory.”
This feature makes it an ideal choice in scientific computing, big data analysis, in-memory databases, and any application scenarios where I/O or memory bandwidth is limited. By efficiently compressing data, Blosc not only saves storage space and network bandwidth but, more importantly, reduces the time taken to move data across memory hierarchies (such as from main memory to CPU cache), thereby accelerating the overall computation process.
2. Core Technologies and Advantages
-
Blocking Blosc divides the input data into smaller blocks (typically tens to hundreds of KB). This blocking strategy brings multiple benefits:
- Cache-friendly: The block size is carefully designed to perfectly match the cache size of modern CPUs, allowing most data processing to occur in the cache, significantly reducing the latency of accessing main memory.
- Parallelization: Each data block can be independently compressed or decompressed, which is the basis for achieving multi-threaded parallel processing.
Multi-threading Support Blosc automatically manages a thread pool internally. With just one parameter, users can specify the number of threads to use for compression/decompression. It efficiently allocates different data blocks to different threads, fully utilizing the parallel computing capabilities of multi-core CPUs, with speed improvements nearly proportional to the number of cores.
As a Meta-compressor Blosc is not entirely a new compression algorithm but a processing framework. It integrates various efficient compression algorithms (referred to as “compression codecs”) and can apply preprocessing programs (called “filters”) to the data before compression.
Supported Compression Codecs: BloscLZ (default), LZ4, LZ4HC, Zstd, Zlib, etc. Users can choose based on their needs for speed and compression ratio (for example, LZ4 is the fastest, while Zstd offers a good balance between speed and ratio).
Shuffle Filter: This is Blosc’s ace feature. For data containing multiple bytes (such as <span>int32</span>, <span>float64</span>), the <span>shuffle</span> filter groups all values’ highest-order bytes, next highest-order bytes, etc. This rearrangement often creates long sequences of repeated bytes in the data, significantly enhancing the compression ratio of subsequent compression codecs, with minimal overhead.
SIMD Acceleration Blosc uses SIMD instruction sets such as SSE2 and AVX2 in critical paths (like the <span>shuffle</span> and <span>bitshuffle</span> filters) for acceleration, further squeezing hardware performance.
3. C++ Code Example
The following example demonstrates how to use Blosc’s C++ API for basic compression and decompression.
Environment Setup: Before compiling, you need to install the Blosc library.
- Linux/macOS: Typically can be installed via package managers, such as
<span>sudo apt-get install libblosc-dev</span>or<span>brew install c-blosc</span>. - Windows: Can be installed via vcpkg or conda.
Compilation Command (example):
g++ -std=c++11 -O2 -o blosc_example blosc_example.cpp -lblosc
Example Code:
#include <iostream>
#include <vector>
#include <blosc.h>
#include <cassert>
int main() {
// 0. Initialize Blosc (optional but recommended)
blosc_init();
// 1. Prepare raw data
// Create a vector containing 1 million floating-point numbers
const size_t num_elements = 1000000;
std::vector<float> original_data;
original_data.reserve(num_elements);
// Generate some patterned data (easy to compress)
for (size_t i = 0; i < num_elements; ++i) {
original_data.push_back(static_cast<float>(i % 100)); // Repeating pattern 0-99
}
size_t original_size = original_data.size() * sizeof(float);
std::cout << "Original data size: " << original_size << " bytes" << std::endl;
// 2. Prepare buffer for compressed output
// In the worst case, compressed data may be slightly larger than original data. Blosc provides a macro to calculate the maximum size.
size_t max_compressed_size = original_size + BLOSC_MAX_OVERHEAD;
std::vector<char> compressed_data(max_compressed_size);
// 3. Perform compression
// blosc_compress(clevel, doshuffle, typesize,
// src, srcsize, dest, destsize)
// clevel: compression level [0-9], 0=no compression, 9=max compression
// doshuffle: whether to enable byte shuffling preprocessing [0, BLOSC_NOSHUFFLE, BLOSC_SHUFFLE, BLOSC_BITSHUFFLE]
// typesize: byte size of a single data element (e.g., sizeof(float))
int clevel = 5; // Medium compression level
int doshuffle = BLOSC_SHUFFLE; // Enable SHUFFLE filter, very effective for floats
int typesize = sizeof(float);
int compressed_size = blosc_compress(
clevel, doshuffle, typesize,
original_data.data(), original_size,
compressed_data.data(), compressed_data.size()
);
if (compressed_size <= 0) {
std::cerr << "Compression failed!" << std::endl;
return 1;
}
std::cout << "Compressed successfully. Size: " << compressed_size << " bytes"
<< " (Ratio: " << (float)original_size / compressed_size << "x)" << std::endl;
// 4. Perform decompression
// First, we need a buffer the same size as the original data
std::vector<float> decompressed_data(num_elements);
size_t decompressed_size = decompressed_data.size() * sizeof(float);
// blosc_decompress(src, dest, destsize)
int size_check = blosc_decompress(
compressed_data.data(), decompressed_data.data(), decompressed_size
);
if (size_check <= 0) {
std::cerr << "Decompression failed!" << std::endl;
return 1;
}
assert(size_check == (int)decompressed_size); // The number of bytes decompressed should match the original data
// 5. Verify that the decompressed data matches the original data exactly
if (original_data == decompressed_data) {
std::cout << "Verification successful: Decompressed data matches original." << std::endl;
} else {
std::cerr << "Verification failed: Data mismatch!" << std::endl;
}
// 6. Clean up Blosc (optional but recommended)
blosc_destroy();
return 0;
}
Code Explanation and Key Points:
- Data Preparation: We created a vector of
<span>float</span>data with a repeating pattern. This structured data is an ideal target for compression algorithms (especially when combined with<span>shuffle</span>). - Buffer Allocation: The size of the compressed output buffer must be at least
<span>original size + BLOSC_MAX_OVERHEAD</span>to account for potential data expansion in case of compression failure. - **Compression Function
<span>blosc_compress</span>**:
<span>clevel</span>: Compression level. The higher the level, the higher the potential compression ratio, but the slower the speed. For speed-focused scenarios, levels 1-3 are common choices.<span>doshuffle</span>:<span>BLOSC_SHUFFLE</span>is very effective for data types like<span>int</span>and<span>float</span>that have fixed byte lengths.<span>typesize</span>: Must be set correctly (e.g.,<span>float</span>is 4,<span>double</span>is 8). This is necessary for the<span>shuffle</span>filter to work properly.
4. Conclusion
Blosc successfully transforms data compression from a performance burden into a performance accelerator through its unique blocking, multi-threading, SIMD, and filtering technologies. When your application is constrained by memory bandwidth or I/O speed, integrating Blosc is a highly effective optimization strategy. Its simple API allows developers to easily incorporate it into data processing pipelines in C, C++, and even Python (via <span>python-blosc</span> or <span>NumExpr</span>).