The 'Ghost' in Rust Compilation: A Peculiar Build Failure Case

Introduction

Have you ever encountered a strange situation during Rust development where running cargo check works perfectly in one directory, but fails to compile in another? Or where the CI environment fails to build, but it passes locally?

Recently, a user of cargo-semver-checks faced such a peculiar issue: the tool reported that it could not build their crate, yet they could always succeed when running cargo check themselves. What could be the reason? Let’s follow the author’s debugging journey, Predrag, to uncover the true nature of this “ghost”.

The Cause of the Problem: The “Not My Fault” Rule

cargo-semver-checks is a tool used to check the semantic version compatibility of Rust projects. To perform the checks, it needs to build the package being checked. When the build fails, the tool generates a “reproduction script” that allows users to independently verify that the issue is indeed with the project itself and not a bug in the tool.

This mechanism is crucial because it can:

Provide a better experience for users
Avoid generating a large number of invalid bug reports in the issue tracker
Save valuable time and effort for maintainers

Before version v0.45, when the build failed, the output looked like this:

error: running cargo-doc on crate 'my_crate' failed with output:
-----
Documenting my_crate v0.1.0 ([PATH]/my_crate)
error: This crate has a compiler error.
--> [PATH]/my_crate/src/lib.rs:6:1
|
6 | compile_error!("This crate has a compiler error.");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: could not document `my_crate`
-----
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path ./my_crate &&
cargo check

However, the issue encountered this time was that when the user ran this reproduction script, they received a successful output!

Debugging Journey: Unraveling the Mystery

The author began a systematic investigation:

First Round of Investigation

cargo-semver-checks version? ✓ Latest v0.44.0
Rust version? ✓ Within support range
Was --target used for cross-compilation? ✗ No
Running environment? CI on Linux, local also on Linux

Second Round of Investigation

Letting the user run cargo-semver-checks on their local machine: it passed! No build failure!

This indicated that the issue was related to the running environment. But both CI and local were Linux, and the Rust versions were the same, so why was there a difference?

Third Round of Investigation

Was the lock file committed to the repository? ✓ Yes
Were there any special dependencies? A key clue was discovered!

The user’s .cargo/config.toml file had set a <code>--cfg flag using build.rustflags, and a certain dependency required this flag to compile.

Key Discovery: Directory Location Matters

Further investigation revealed an “interesting” phenomenon:

Running cargo check within the project directory: ✓ Success
Running with --manifest-path from another directory: ✗ Failure

The reason lies in how cargo searches for configuration files. According to the Cargo Book, cargo starts searching for configuration files from the current working directory, not from the project directory specified by --manifest-path.

Example of search order (assuming cargo is called from the /projects/foo/bar/baz directory):

/projects/foo/bar/baz/.cargo/config.toml
/projects/foo/bar/.cargo/config.toml
/projects/foo/.cargo/config.toml
/projects/.cargo/config.toml
/.cargo/config.toml
$CARGO_HOME/config.toml  # defaults to $HOME/.cargo/config.toml

When using --manifest-path from outside the project directory, cargo cannot find the project's <code>.cargo/config.toml, thus missing the required <code>--cfg flag, leading to the compilation failure.

The Real Culprit: cargo doc ≠ cargo check

The story isn’t over yet. A closer look at the compilation error reveals that even running cargo-semver-checks within the project directory fails. Why?

The key is:cargo-semver-checks actually runs cargo doc, not <code>cargo check!

The difference between these two commands is crucial:

cargo check / cargo build

Interacts with rustc
Reads RUSTFLAGS environment variable
Reads build.rustflags configuration

cargo doc

Runs rustdoc (though it also uses rustc under the hood)
Reads RUSTDOCFLAGS environment variable
Reads build.rustdocflags configuration

The user’s configuration file only set build.rustflags, but did not set <code>build.rustdocflags! Therefore:

cargo check can pass (with the <code>--cfg flag)
cargo doc fails (missing the <code>--cfg flag)

Reproducing the Case

To help everyone better understand this issue, the author created a test repository. We can build a simplified reproduction case:

Step 1: Create a dependency that requires a specific cfg flag

// dependency/src/lib.rs
#[cfg(not(feature = "special_flag"))]
compile_error!("The special_flag must be enabled to compile");

pub fn hello() {
    println!("Hello from dependency!");
}

Step 2: Create the main project and configure .cargo/config.toml

# .cargo/config.toml
[build]
rustflags = ["--cfg", "feature=\"special_flag\""]
# Note: rustdocflags is not set here

Step 3: Test different build commands

# Within the project directory
$ cargo check
   Compiling dependency v0.1.0
   Compiling my_project v0.1.0
    Finished dev [unoptimized + debuginfo] target(s) in 0.50s

# Generate documentation
$ cargo doc
   Documenting dependency v0.1.0
error: The special_flag must be enabled to compile
 --> dependency/src/lib.rs:2:1
  |
2 | compile_error!("The special_flag must be enabled to compile");
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

# Using --manifest-path from another directory
$ cd ~
$ cargo check --manifest-path /path/to/project/Cargo.toml
error: The special_flag must be enabled to compile

Solution

cargo-semver-checks v0.45 introduced two improvements:

1. Correctness Improvements

The tool can now:

Replicate cargo’s configuration search process
Detect and pass any encountered RUSTFLAGS or <code>RUSTDOCFLAGS
Instead of simply overriding them with its own configuration

This implementation is quite complex, as these values can be specified in 6 different environment variables or 5 different configuration keys, including target-specific ways. All of this is supported by 28 new test cases covering various peculiar crate and configuration combinations.

2. User Experience Improvements

The “Not My Fault” script is now smarter:

It suggests running both cargo check and <code>cargo doc
Includes the --target flag when necessary to select the correct target platform

Example of improved error message:

error: failed to build rustdoc for crate my_crate v0.1.0
note: this is usually due to a compilation error in the crate
note: you can reproduce this error with:
  cargo check --manifest-path /path/to/Cargo.toml
  cargo doc --manifest-path /path/to/Cargo.toml
note: if using a custom target, add: --target your-target-triple

Best Practice Recommendations

From this case, we can summarize some best practices:

1. Integrity of Configuration Files

If you set build.rustflags in <code>.cargo/config.toml, consider also setting <code>build.rustdocflags:

[build]
rustflags = ["--cfg", "feature=\"my_flag\""]
rustdocflags = ["--cfg", "feature=\"my_flag\""]

2. Thoroughness of Testing

Don’t just test cargo build or <code>cargo check, also test:

cargo doc: Ensure documentation can be generated correctly
cargo test: Tests may use different compilation configurations
cargo bench: Benchmarking may also expose issues

3. Consistency in CI Environments

In CI, try to simulate the local development environment:

# .github/workflows/ci.yml
- name: Check
  run: |
    # Run within the project directory
    cd ${{ github.workspace }}
    cargo check
    cargo doc --no-deps
    cargo test

4. Caution with –manifest-path

If you need to use --manifest-path, ensure you run from the correct directory or use absolute paths:

# Bad practice (may not find configuration)
cd ~
cargo check --manifest-path /path/to/project/Cargo.toml

# Good practice
cd /path/to/project
cargo check

# Or explicitly set the working directory
(cd /path/to/project && cargo check)

Deep Understanding: The Hierarchy of Rust Compilation Configuration

This issue reveals a complex yet important feature of the Rust build system: the hierarchy of configurations.

Configuration Priority

From highest to lowest:

Command line arguments (e.g., RUSTFLAGS=...)
Environment variables (RUSTFLAGS, <code>RUSTDOCFLAGS)
Target-specific configurations ([target.x86_64-unknown-linux-gnu])
Build configurations ([build])
Defaults

Search Order for Configuration Locations

Current working directory/.cargo/config.toml
↓
Parent directory/.cargo/config.toml
↓
Grandparent directory/.cargo/config.toml
↓
...upwards...
↓
Root directory/.cargo/config.toml
↓
$CARGO_HOME/config.toml (usually ~/.cargo/config.toml)

Complete Configuration Example

# .cargo/config.toml

# Basic build configuration
[build]
# These flags are passed to rustc
rustflags = [
    "--cfg", "feature=\"custom\"",
    "-C", "target-cpu=native"
]
# These flags are passed to rustdoc
rustdocflags = [
    "--cfg", "feature=\"custom\"",
    "--cfg", "docsrs"
]

# Target-specific configuration
[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

# Aliases for convenience
[alias]
check-all = "check --all-features --all-targets"
doc-all = "doc --all-features --no-deps"

Conclusion

This “ghost” case illustrates:

The complexity of the Rust build system: Even experienced developers can encounter unexpected issues
The importance of configuration lookup: cargo searches for configurations from the working directory rather than the project directory, which is easily overlooked
The differences between cargo check and cargo doc: They use different environment variables and configuration keys
The necessity of comprehensive testing: Not only should compilation be tested, but also documentation generation
Considerations in tool design: Good error messages should help users quickly locate issues

As the author states, maintaining open-source projects is a continuous learning process. Each new version is an opportunity for progress, and each bug is a chance to gain a deeper understanding of the system. Although this issue was peculiar, through patient debugging, detailed test cases, and a focus on user experience, a solution was ultimately found.

For Rust learners, this case reminds us:

Understanding how tools work is more important than just using them
When encountering problems, systematic investigation is more effective than blind attempts
Complex systems require complex testing to ensure correctness

We hope this article helps you quickly locate and resolve similar issues in the future!

References

Ghosts in the Compilation: https://predr.ag/blog/ghosts-in-the-compilation/

Book Recommendations

The second edition of “The Rust Programming Language” is an authoritative learning resource written by the Rust core development team and translated by members of the Chinese Rust community. It is suitable for all software developers who wish to evaluate, get started, improve, and study the Rust language, and is regarded as essential reading for Rust development work.

This book introduces the fundamental concepts of the Rust language to unique practical tools, covering advanced concepts such as ownership, traits, lifetimes, and safety guarantees, as well as practical tools like pattern matching, error handling, package management, functional features, and concurrency mechanisms. The book includes three complete project development case studies, guiding readers from zero to developing Rust practical projects.

Notably, this book has been updated to include content from the Rust 2021 edition, meeting the systematic learning needs of beginners and serving as a reference guide for experienced developers, making it the best entry point for building solid Rust skills.

The ‘Ghost’ in Rust Compilation: A Peculiar Build Failure Case

Introduction

The Cause of the Problem: The “Not My Fault” Rule

Debugging Journey: Unraveling the Mystery

Key Discovery: Directory Location Matters

The Real Culprit: cargo doc ≠ cargo check

Reproducing the Case

Solution

1. Correctness Improvements

2. User Experience Improvements

Best Practice Recommendations

1. Integrity of Configuration Files

2. Thoroughness of Testing

3. Consistency in CI Environments

4. Caution with –manifest-path

Deep Understanding: The Hierarchy of Rust Compilation Configuration

Configuration Priority

Search Order for Configuration Locations

Complete Configuration Example

Conclusion

References

Book Recommendations

Leave a Comment Cancel reply

Introduction

The Cause of the Problem: The “Not My Fault” Rule

Debugging Journey: Unraveling the Mystery

Key Discovery: Directory Location Matters

The Real Culprit: cargo doc ≠ cargo check

Reproducing the Case

Solution

1. Correctness Improvements

2. User Experience Improvements

Best Practice Recommendations

1. Integrity of Configuration Files

2. Thoroughness of Testing

3. Consistency in CI Environments

4. Caution with –manifest-path

Deep Understanding: The Hierarchy of Rust Compilation Configuration

Configuration Priority

Search Order for Configuration Locations

Complete Configuration Example

Conclusion

References

Book Recommendations

Related posts

Leave a Comment Cancel reply