Cross Compilation

In the vehicle scenario, GreptimeDB Edge is usually deployed as a service in the Android environment, requiring us to compile it into an executable file suitable for the Android platform. A preliminary solution might be to purchase an Android development board and install the Rust toolchain for compilation. However, this approach may face the following challenges:

1. Configuring the Rust compilation environment on the Android development board can be complex (the author has not configured it in practice);
2. Most Android development boards have weak CPU performance, making the compilation of large projects slow and inefficient;
3. Local Android API versions may differ from the API versions on the target device, and even the CPU architecture may vary, leading to compatibility issues.

Relatively speaking, cross-compilation provides a more efficient alternative. It allows developers to compile programs that can run on one system platform (such as x86 PCs) from another system platform (such as ARM mobile devices), which is especially useful when direct compilation on the target system is challenging.

Rust has excellent support for cross-compilation, and the Android NDK provides the necessary toolchains and libraries, further simplifying the cross-compilation process. Since our development or compilation environment is usually macOS or Linux, choosing to generate Android executable files through cross-compilation is an ideal solution.

Rust Compilation

First, we need to have a general understanding of the Rust compilation process. Rustc first compiles Rust code into LLVM-IR, and then LLVM compiles LLVM-IR into binaries for each platform, which are finally linked together by the linker to generate the final binary file.

❝

Rustc is the compiler for Rust, using LLVM as the backend (it can also be said that Rustc is the frontend for LLVM).

Below is a simplified version of the Rust compilation architecture diagram:

GreptimeDB Cross Compilation Practice

GreptimeDB Edge is built on the open-source version of GreptimeDB. Therefore, we will take the open-source version of GreptimeDB[1] as an example and demonstrate step by step how to perform cross-compilation on x86 Linux to generate an executable file for the aarch64-linux-android architecture.

First, install the Android NDK, which can be downloaded from https://developer.android.com/ndk/downloads?hl=en. Additionally, set an environment variable for convenience in subsequent operations, as shown below:

export ANDROID_NDK_HOME=<YOUR_NDK_ROOT>

# Example
# export ANDROID_NDK_HOME=/home/fys/soft/ndk/android-ndk-r25c

Next, pull the source code of GreptimeDB from GitHub:

git clone https://github.com/GreptimeTeam/greptimedb.git --depth 1

Then, adding the target to the Rust toolchain is a key step in achieving cross-platform compilation. This allows Rustc to compile the intermediate representation LLVM-IR code into the machine language of the target platform. In this example, the target platform architecture is aarch64-linux-android, and execute the following command in the root directory of the GreptimeDB project:

rustup target add aarch64-linux-android

❝

Rust platform support see here[2].

At this point, attempting to compile may result in an error: “-lgcc” not found. The reason is that the Android NDK’s libgcc.a has been replaced by libunwind.a. The solution is to copy libunwind.a and rename it to libgcc.a, see Rust blog[3].

# The specific path may vary with different versions of ndk.
cd $ANDROID_NDK_HOME/toolchains/llvm/prebuilt/linux-x86_64/lib/clang/17/lib/linux/aarch64/
cp libunwind.a libgcc.a

In projects that only involve the Rust language, developers typically need to configure the linker and archiver. However, Rust supports executing build tasks in build scripts (build.rs), so during the compilation process of a Rust project, it may be necessary to integrate compilation work in other languages such as C and C++. This often requires providing some necessary information to the compilation tools (such as cc[4] or cmake[5], including compiler paths (CC and CXX), library files, and header file locations, etc. This process can often be quite complex.

cargo-ndk[6] This project helps us solve most of the issues. By executing the following command, we can compile the GreptimeDB binary program suitable for the aarch64-linux-android platform.

cargo ndk --platform 30 -t aarch64-linux-android build --bin greptime --release

Additionally, for libraries that are not compatible with specific target platforms, handling them can indeed be tricky. One solution is to replace them with compatible libraries; if the functionality is not essential, feature guards can be used to remove them during the compilation phase.

❝

During the compilation process, if you encounter an error indicating that the protobuf library is missing or other issues, simply install it correctly.

Common Issues

Previously, I encountered a problem where enabling LTO[7] optimization caused the cross-compilation of GreptimeDB to fail. The error message is as follows:

  = note: ld.lld: error: duplicate symbol: pthread_atfork
          >>> defined at crtbegin.c
          >>>            /home/fys/soft/ndk/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/lib/aarch64-linux-android/23/crtbegin_dynamic.o:(pthread_atfork)
          >>>            defined at build_jemalloc.6cd863fbc26b10-cgu.0
          >>>            /home/fys/source/build_jemalloc/target/aarch64-linux-android/nightly/deps/build_jemalloc-c1434931e7fc5ee2.build_jemalloc.6cd863fbc26b10-cgu.0.rcgu.o:(.text.pthread_atfork+0x0)
          clang-17: error: linker command failed with exit code 1 (use -v to see invocation)

When API level >= 21, Android provides a declaration for pthread_atfork. The tikv-jemallocator[7] also has a declaration for pthread_atfork, both are strong symbol types, and when LTO optimization is enabled, it leads to symbol conflicts. The solution is to set pthread_atfork in tikv-jemallocator to weak symbol type.

The latest version of tikv-jemallocator has resolved this issue, see here[8].

Backtrace on Android

During the development of the GreptimeDB Edge project, we observed that the standard library of the Rust language does not provide the expected stack trace information in the Android environment. Specifically, when the program panic, the relevant stack information is not correctly captured and instead shows as unknown, which greatly complicates problem diagnosis.

Reproducing the Issue

To reproduce this issue, we wrote a simplified example program.

1. In the main method, we triggered a panic to simulate an exception:

fn main() {
    panic!("Panic here.");
}

1. Specify rust-toolchain as stable 1.81 or a lower version:

[toolchain]
channel = "1.81"

1. Cross-compile to generate binaries that run on Android. During this process, we can review and reinforce the content of the previous section:

export ANDROID_NDK_HOME=<YOUR_NDK_ROOT>
rustup target add aarch64-linux-android
cargo ndk --platform 28 -t aarch64-linux-android build --release

1. Push the binary to the Android virtual machine and execute:

RUST_BACKTRACE=full ./<binary path>

1. The execution result indicates that the expected backtrace information was not successfully obtained. The issue has been reproduced!

thread 'main' panicked at src/main.rs:2:5:
        Panic here
        stack backtrace:
           0:     0x5d908f7a7535 - <unknown>
           1:     0x5d908f7b336b - <unknown>
           2:     0x5d908f7a617f - <unknown>
           3:     0x5d908f7a8541 - <unknown>
           4:     0x5d908f7a817e - <unknown>
           5:     0x5d908f7a8fe8 - <unknown>
           6:     0x5d908f7a8ed3 - <unknown>
           7:     0x5d908f7a7a09 - <unknown>
           8:     0x5d908f7a8b94 - <unknown>
           9:     0x5d908f7b2753 - <unknown>
          10:     0x5d908f7a1e0c - <unknown>
          11:     0x5d908f7a1db3 - <unknown>
          12:     0x5d908f7a1da9 - <unknown>
          13:     0x5d908f7a4eb9 - <unknown>
          14:     0x5d908f7a1e35 - <unknown>
          15:     0x7d6d9c64478d - <unknown>

Solution

We first introduce the solution so that those who are not interested in the cause can skip the next section.

Upgrade the Rust toolchain version: It is recommended to upgrade the rust-toolchain version to 1.82 or higher. This issue has been fixed in version 1.82 (the next section will introduce the fix method).

Custom Panic Hook: Rust supports registering a custom panic hook function to replace the default behavior. If upgrading the Rust version is not possible, you can use the backtrace-rs library to set a custom panic hook function.

❝

The default panic hook function in Rust may not meet the needs in specific environments. For example, on the Android platform, you might prefer to output panic information to a file or logcat, while the default panic hook function only outputs panic information to standard error. Therefore, in many scenarios, we need to customize the panic hook function.

Below is an implementation example:

pub fn set_panic_hook() {
    #[cfg(windows)]
    const LINE_ENDING: &str = "\r\n";
    #[cfg(not(windows))]
    const LINE_ENDING: &str = "\n";

    std::panic::set_hook(Box::new(move |panic| {
        let backtrace = backtrace::Backtrace::new();

        let Some(l) = panic.location() else {
            log::error!(
                "Panic: {:?}, backtrace: {}{:#?}",
                panic, LINE_ENDING, backtrace
            );
            return;
        };

        log::error!(
            "Panic: {:?}, file: {}, line: {}, col: {}, backtrace: {}{:#?}",
            panic,
            l.file(),
            l.line(),
            l.column(),
            LINE_ENDING,
            backtrace,
        );
    }));
}

The output stack information is as follows (the compilation options have removed debug info, while retaining the symbol table):

Panic: PanicHookInfo { payload: Any { .. }, location: Location { file: "src/main.rs", line: 3, col: 5 }, can_unwind: true, force_no_backtrace: false }, file: src/main.rs, line: 3, col: 5, backtrace:
   0:     0x5a58805bdf63 - cross_compile_on_android::set_panic_hook::{{closure}}::h8ff538cfa624b522
   1:     0x5a58805fb0f3 - std::panicking::rust_panic_with_hook::h1f4c9072872fa4b1
   2:     0x5a58805fadb3 - std::panicking::begin_panic_handler::{{closure}}::h73465221de2f2f04
   3:     0x5a58805f9689 - std::sys::backtrace::__rust_end_short_backtrace::h67f67f7cadadf1c3
   4:     0x5a58805faa74 - rust_begin_unwind
   5:     0x5a5880605b63 - core::panicking::panic_fmt::h394cd2a8b9d0c24
   6:     0x5a58805bdf11 - cross_compile_on_android::main::h477274cf7246f129
   7:     0x5a58805bdb53 - std::sys::backtrace::__rust_begin_short_backtrace::hb593986c2bdf2ffe
   8:     0x5a58805bdb49 - std::rt::lang_start::{{closure}}::hdba3573990c4d5eb
   9:     0x5a58805f4519 - std::rt::lang_start_internal::h50565391ca281790
  10:     0x5a58805be1f5 - main
  11:     0x7e5b4020278d - __libc_init

❝

Note: The output stack information also relates to compilation options. If both the symbol table and debug info are removed from the binary, it will generate unknown stacks. If debug info is retained, the stack information will be more detailed, but the binary size will increase significantly.

Cause of the Issue

Next, we will explore the previously raised issue based on Rust 1.81.

Prerequisite Knowledge

1. The backtrace of the Rust standard library relies on the backtrace-rs[10] library, which is integrated into the Rust standard library as a git submodule, see here[11].

2. backtrace-rs[12] checks the Android API version during the build process. If it is greater than or equal to 21, it enables the dl_iterate_phdr feature. see here[13] (Note: the version of backtrace-rs is the version that Rust 1.81 depends on, not the latest version).

Combining the above two points, the Rust standard library introduces backtrace-rs in the form of a git submodule, but does not execute the build logic in backtrace-rs, causing the dl_iterate_phdr feature to not be enabled. Therefore, the standard library’s backtrace cannot function properly on Android.

Case solved!

Solution

In fact, we only need to enable the dl_iterate_phdr feature in the standard library for backtrace-rs. However, starting from #120593, Rust has raised the minimum supported API version for Android from 19 to 21, and from 21 onwards, Android supports dl_iterate_phdr[15] More information can be foundhere[16]. Therefore, we can directly enable the dl_iterate_phdr feature in the backtrace-rs library without needing to check the Android API version (Rust 1.82 has also fixed this).

Conclusion

Cross-compilation has always been very tricky, and various issues can arise without fixed solutions; we always have to address specific problems. Fortunately, Cargo NDK and Android NDK provide a convenient solution to help us effectively deal with most compilation issues.

Through this discussion, we recognize the importance of cross-compilation in the Android environment and the advantages of the Rust compilation mechanism. Although the ideal compilation process faces numerous challenges in practice, we hope our experiences can provide practical references and inspiration for future development.

Reference:

[1] https://github.com/GreptimeTeam/greptimedb[2] https://doc.rust-lang.org/nightly/rustc/platform-support.html[3] https://blog.rust-lang.org/2023/01/09/android-ndk-update-r25.html[4] https://github.com/rust-lang/cc-rs[5] https://github.com/rust-lang/cmake-rs[6] https://github.com/bbqsrc/cargo-ndk[7] https://doc.rust-lang.org/cargo/reference/profiles.html#lto[8] https://github.com/tikv/jemallocator/issues/81[9] https://github.com/tikv/jemallocator[10] https://doc.rust-lang.org/1.81.0/std/backtrace/index.html[11] https://docs.rs/backtrace/latest/backtrace/ [12] https://github.com/rust-lang/rust/blob/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/lib.rs#L667-L669[13] https://docs.rs/backtrace/latest/backtrace/ https://github.com/rust-lang/backtrace-rs/blob/72265bea210891ae47bbe6d4f17b493ef0606619/build.rs#L52-L54[14] https://github.com/rust-lang/rust/pull/120593 [15] https://man7.org/linux/man-pages/man3/dl_iterate_phdr.3.html [16] https://android.googlesource.com/platform/bionic/+/HEAD/docs/status.md

[17] https://github.com/rust-lang/backtrace-rs/pull/656

[18] https://github.com/rust-lang/rust/pull/129305

❝

About Greptime

Greptime focuses on providing real-time and efficient data storage and analysis services for observable, IoT, and vehicle networking fields, helping clients tap into the deep value of data. Currently, the cloud-native time-series database GreptimeDB has derived several solutions suitable for different users. For more information or demo presentations, please contact the assistant below (WeChat ID: greptime).

We welcome friends interested in open source to contribute and discuss, starting your open-source journey from issues tagged with good first issue! We look forward to meeting you in the open-source community! Add the assistant on WeChat to join the “Technical Exchange Group” for face-to-face communication with like-minded friends!

Star us on GitHub Now: https://github.com/GreptimeTeam/greptimedbOfficial website:https://greptime.cn/Documentation:https://docs.greptime.cn/Twitter:https://twitter.com/GreptimeSlack:https://greptime.com/slackLinkedIn:https://www.linkedin.com/company/greptime/

Previous wonderful articles:

Click “Read the original text” to experience GreptimeDB now!