Reducing Rust Compilation Time from 30 Minutes to 2 Minutes: The Challenge of Handling a Thousand Libraries
Rust is very fast at runtime—but not so much at compile time. This is no news to anyone with serious experience writing Rust code. There is a whole category of blog posts dedicated to reducing cargo build times.
At Feldera, we allow users to define tables and views through SQL. Behind the scenes, we convert this SQL into Rust code—then compile it into a standalone binary using rustc, which gradually maintains all views as new data flows into the tables.
To speed up compilation, many methods have been tried: type erasure, aggressive code deduplication, limiting the number of lines of code generated, etc. These measures have yielded good results. However, we recently started onboarding a new large enterprise client whose SQL is quite complex. For example, one of the programs they wrote consists of 8,562 lines of SQL code, which the Feldera SQL-to-Rust compiler translated into approximately 100,000 lines of Rust code.
To clarify, this is not about compiling some massive monolithic application. We are talking about around 100,000 lines of generated Rust code. Compared to the Linux kernel (which contains 40 million lines of code but compiles in just a few minutes), this is trivial.
Even so… this program took about 25 minutes to compile on my machine. Worse yet, in the client’s setup, it took about 45 minutes. And this was after already optimizing the generated code.
Here are the logs from the Feldera manager:
[manager] SQL compilation success: pipeline 0196268e-7f98-7de3-b728-0ee339e449fa (program version: 2) (took 101.94s)
[manager] Rust compilation success: pipeline 0196268e-7f98-7de3-b728-0ee339e449fa (program version: 2) (took 1617.77s; source checksum: cbffcb959174; integrity checksum: 709a17251475)
Almost all the time was spent on Rust compilation. Converting SQL to Rust only took about 1 minute and 40 seconds. Worse still, the Rust build is equivalent to a release build using cargo, so it starts from scratch every time (except for those cargo crate dependencies that have already been cached/reused). Even a minor change in the input SQL triggers a complete rebuild of the entire large program.
Of course, debug builds have also been attempted. While they can reduce the time to about 5 minutes, they are not practical in practice. The client cares about actual runtime performance: when the SQL code passes type checking, they already know the Rust code will compile successfully, and they are running a real-time data pipeline, hoping to see end-to-end latency and throughput. Debug builds are too slow and misleading for this purpose.
So where is the problem?
Frustratingly, even though we are using a machine with a 64-core processor and 128 threads, Rust hardly utilizes these resources. This becomes evident when looking at htop during the compilation process:
Indeed, only one core is fully utilized while the others are “asleep”.
By passing the -Ztime-passes flag to RUSTFLAGS to inspect the compilation process of this crate (which requires recompiling with the nightly version), it was found that most of the time was spent on LLVM passes and codegen—unfortunately, they are all single-threaded.
Sometimes during those 30 minutes, Rust would start a few threads—maybe 3 or 4—but it never fully utilized the machine’s capabilities. Far from it.
It is clear: parallelizing compilation is indeed fraught with difficulties. But this is not an edge case; there are clearly enough opportunities to parallelize the compilation process of this program upon personal observation.
What is the solution? Instead of emitting a huge crate containing everything, we adjusted the SQL-to-Rust compiler to split the output into many smaller crates. Each crate encapsulates a portion of the logic and depends on each other, pulled together by a top-level main crate.
The results were astonishing. Here is what the htop view of the modified compilation process looks like:
Wonderful. All CPUs are now fully utilized.
This reflects a reduction in the time required to compile the Rust program to 2 minutes and 10 seconds!
[manager] Rust compilation success: pipeline 01962739-79fd-7f03-bbf2-f8e29ce21e1d (program version: 2) (took 150.24s; source checksum: 0336f3eb9dc1; integrity checksum: 6051bcde6674)
In most Rust projects, distributing logic across dozens (or even hundreds) of crates is either difficult to achieve or simply a nightmare. But here, it is surprisingly simple—thanks to how Feldera works.
When users write SQL in Feldera, this code is transformed into a data flow graph: nodes are operators that transform data, and edges represent how data flows between them. Here is a snippet of such a graph:
Since the Rust code is entirely generated from this structure, we have complete control over how to split it. Each operator has its own crate. Each crate exports a function to build a specific part of the data flow, following the same pattern. The top-level main crate simply connects them together.
pub fn create_operator_0097dd9de75ffef3(circuit: &RootCircuit, catalog: &mut Catalog,
i0: &Stream<RootCircuit, IndexedWSet<Tup1<i32>, Tup5<i32, SqlString, F64, F64, Option<i32>>>,
i1: &Stream<RootCircuit, IndexedWSet<Tup1<i32>, Tup0>>,
) -> Stream<RootCircuit, WSet<Tup5<i32, SqlString, F64, F64, Option<i32>>> {
// Implementation details omitted...
}
To name these crates, a simple yet powerful method was adopted: hashing the Rust code they contain and using that as the crate name. This ensures two things:First, each crate’s name is unique; more importantly, second, small changes to SQL become extremely efficient. Imagine what happens when a user slightly adjusts their SQL code: most operators (and their crates) remain unchanged (the hash value remains the same), allowing rustc to reuse previously compiled results. Any new code introduced due to changes generates a new crate (with a different hash value).
So, how many crates are there for that massive SQL program? Let’s take a look at the compiler directory inside the Feldera container:
ubuntu@12e1de52de1b:~/.feldera/compiler/rust-compilation$ ls crates/
# Shows examples of multiple crates...
Then execute:
ubuntu@12e1de52de1b:~/.feldera/compiler/rust-compilation$ ls crates/ | wc -l
1106
That’s right, a total of 1,106 crates! Sounds like a lot? Perhaps. But ultimately, this makes rustc more efficient.
But that’s not all. Despite these optimizations, using 128 threads or 64 cores, the entire compilation process should theoretically scale down to about 12 seconds (25 minutes divided by 128), considering hyper-threading it might be around 24 seconds. However, it actually took 170 seconds to complete the entire compilation process. While one cannot expect a linear speedup in practice, being 7 times slower seems excessive (these are independent parallel rustc calls). Similar slowdowns also occur on laptops with fewer memory and cores, so this is not just a large machine issue.
Here are some thoughts on possible reasons:
-
Competition for hardware resources (the system has enough memory, but there may be competition in caching)
-
File system bottlenecks (despite attempts to run on RAM-FS, there was no improvement, but it may relate to file system code locks in the kernel)
-
Linking becoming a new bottleneck? Using mold as the linker, the total linking time is only about 7 seconds.
By changing the way Rust code is generated, we have allowed Feldera’s compilation time to scale with hardware performance rather than against it. Tasks that previously took 30 to 45 minutes to complete can now be done in under 3 minutes, even for complex enterprise-level SQL.
「Click 👇 to follow」
Reference link: https://www.feldera.com/blog/cutting-down-rust-compile-times-from-30-to-2-minutes-with-one-thousand-crates