From Python to Rust: A Developer’s Journey of Performance Awakening

Introduction

As a developer, have you ever found yourself deeply immersed in the gentle embrace of Python? Its concise syntax, rich libraries, and rapid development efficiency seem to solve all problems. However, when your project starts handling gigabytes of data and your API faces high concurrency requests, the limitations of Python begin to surface.

This article shares the journey of a developer transitioning from a “devout Python believer” to a “bilingual Rust user.” This is not an article urging you to abandon Python, but rather to help you understand when to use Python, when to use Rust, and how to perfectly combine the two.

1. Performance Bottlenecks: The Beginning of the Problem

1.1 Encountering Performance Dilemmas

The story begins with a seemingly simple task: processing a CSV file of several gigabytes. The author tried various Python optimization strategies:

• Optimizing loop logic
• Using generators to reduce memory usage
• Switching to Pandas for data processing

However, the script’s runtime remained unacceptable. This prompted the author to ponder, “Why is my Python code so slow?”

1.2 First Encounter with Rust

During the search for answers, the name Rust frequently appeared. The developer community was discussing its three main features:

• Extreme Performance: Close to C/C++ execution speed
• Memory Safety: Most memory errors can be detected at compile time
• Precise Control: Allows developers to master every detail of program execution

Initially, the author thought this was just hype. But one experiment changed everything.

2. The First Rust Experiment: Shocking Results

2.1 Simple Comparison Test

The author decided to rewrite a simple function from the Python script in Rust: reading a CSV file and counting unique values.

Python Implementation:

# Python implementation: concise and intuitive
data = set()
for line in open('data.csv'):
    data.add(line.strip())  # Remove whitespace and add to set
print(len(data))  # Output the number of unique values

This code took about 20 seconds to run.

Rust Implementation Challenge:

The Rust implementation was much more complex. The author needed to:

• Explicitly declare variable types
• Handle potential error cases
• Understand ownership and borrowing mechanisms
• Manage memory allocation

The first five compilations failed. But when the code finally ran, it took less than 2 seconds.

2.2 The Essence of Performance Differences

This tenfold performance improvement was not coincidental. The underlying reasons include:

1. Compiled vs Interpreted: Rust code is compiled into machine code for direct execution, while Python requires an interpreter to execute line by line
2. Memory Management: Rust’s zero-cost abstractions and precise memory control
3. Concurrency Safety: Rust’s type system ensures thread safety, allowing full utilization of multi-core CPUs

3. Application in Real Projects: Image Processing API

3.1 Real-World Scenario

The author was building a Python API using FastAPI, which included image upload, scaling, and storage to AWS. This module performed poorly when processing large batches of images:

• Long response times
• High memory usage
• Occasional memory errors leading to crashes

3.2 Rust Refactoring Solution

The author only rewrote the core image processing module, retaining the FastAPI framework and other Python code. The results were surprising:

• Performance Improvement: Running speed increased nearly tenfold
• Memory Optimization: Memory usage significantly reduced
• Stability Improvement: No more random memory errors

3.3 Key Takeaway: Compile-Time Guarantees

The most important takeaway was not performance, but confidence. Rust’s compiler captures almost all potential issues before the code runs:

• Memory leaks
• Null pointer dereferences
• Data races
• Type mismatches

Errors in Python that only surface at runtime cannot even compile in Rust.

4. Python Still Has Its Value

4.1 Comparison of Development Efficiency

The author emphasizes that he has not abandoned Python. In the following scenarios, Python remains the first choice:

• Rapid Prototyping: Validate an idea in minutes
• Automation Scripts: Concise syntax is suitable for writing operational tools
• AI and Data Science: Rich ecosystem (TensorFlow, PyTorch, Pandas, etc.)
• Glue Code: Connecting different systems and services

In contrast, implementing the same functionality in Rust could take hours.

4.2 Differences in Abstraction Levels

The author used a vivid analogy:

Python is like an automatic car—easy to use, just get in and drive

Rust is like a manual sports car—requires understanding every detail, but offers powerful performance

Python hides many low-level details, allowing you to focus on business logic. Rust, on the other hand, requires you to understand:

• How memory is allocated and released
• Who owns the data
• The lifespan of references

This “forced thinking” actually made the author a better developer.

4.3 Skill Transfer Effects

After learning Rust, the author found that the quality of his Python code also improved:

• More attention to resource management
• Reduced unnecessary copies
• Clearer understanding of program execution flow
• Easier identification of performance bottlenecks

5. Best Practices: Python + Rust Hybrid Architecture

5.1 Decision Tree for Language Selection

After several months of practice, the author summarized a set of practical decision principles:

Scenarios for Using Python:

• Need for rapid development and iteration
• Performance requirements are not high
• Dependent on specific libraries in the Python ecosystem
• Team members are more familiar with Python

Scenarios for Using Rust:

• Performance-critical paths (CPU-intensive tasks)
• Need to ensure memory safety
• System-level programming
• Need precise control over resource usage

5.2 Hybrid Development Solutions

The most powerful solution is to combine both:

1. Build the application framework with Python: API layer, business logic, database interaction
2. Implement performance-critical modules with Rust: Data processing, image/video encoding/decoding, encryption algorithms
3. Connect the two via FFI or PyO3

Simple Example Using PyO3

Rust Side (lib.rs):

use pyo3::prelude::*;

/// High-performance data processing function implemented in Rust
#[pyfunction]
fn process_large_data(data: Vec<String>) -> PyResult<usize> {
    // Rust's efficient processing logic
    let unique_count = data.iter()
        .map(|s| s.trim())  // Remove whitespace characters
        .collect::<std::collections::HashSet<_>>()  // Collect into HashSet for deduplication
        .len();  // Return the number of unique values
    
    Ok(unique_count)
}

/// Python module definition
#[pymodule]
fn rust_module(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(process_large_data, m)?)?;
    Ok(())
}

Python Side Call:

import rust_module

# Prepare a large dataset
large_dataset = ["item1", "item2", "item1", "item3"] * 1000000

# Call the high-performance function implemented in Rust
unique_count = rust_module.process_large_data(large_dataset)
print(f"Number of unique values: {unique_count}")

5.3 Example of Project Architecture

Taking a Web API as an example:

┌─────────────────────────────────────┐
│     FastAPI / Flask (Python)        │  ← API routing, authentication, business logic
├─────────────────────────────────────┤
│   Middleware (PyO3 / ctypes)        │  ← Bridge between Python and Rust
├─────────────────────────────────────┤
│   Performance-Critical Module (Rust) │  ← Image processing, data parsing, encryption
└─────────────────────────────────────┘
         │
         ▼
   Database / Cache / External Services

6. Suggestions for Learning Rust

6.1 Mindset Adjustment

Transitioning from Python to Rust is not an overnight process. The author’s suggestions include:

• Expect Compilation Errors: Frequent compilation failures in the early stages are normal
• Embrace the Compiler: Treat the compiler’s error messages as your best teacher
• Take Small Steps: Start by rewriting small functions and gradually accumulate experience
• Don’t Force It: Not all code needs to be rewritten in Rust

6.2 Recommended Learning Path

1. Basic Syntax: Ownership, borrowing, lifetimes
2. Common Data Structures: Vec, HashMap, Option, Result
3. Error Handling: The ? operator and custom error types
4. Concurrent Programming: Threads, channels, Arc, Mutex
5. Practical Projects: Choose a real problem to solve with Rust

6.3 Common Pitfalls

Pitfall 1: Over-Optimization

Don’t get caught up in performance optimization from the start. Get the code running first, then optimize hotspots based on profiling results.

Pitfall 2: Fighting the Compiler

If the compiler rejects your code, it’s likely a design issue, not that the compiler is too strict.

Pitfall 3: Ignoring the Ecosystem

Rust’s library ecosystem is not as mature as Python’s, and certain areas may lack ready-made solutions.

7. Core Insights

At the end of the article, the author shares the most important cognitive shift:

I used to believe that one language could solve all problems.

After learning Rust, I realized that every language has its purpose.

Python gives you freedom, while Rust gives you control.

When you master both, you no longer code blindly—you begin to truly understand everything happening behind the screen.

This is not just about tool selection, but about upgrading your mindset:

• From “as long as it runs” to “understanding the principles”
• From “quick implementation” to “long-term maintenance”
• From “single tool” to “toolbox thinking”

Conclusion

Python and Rust are not mutually exclusive; they are complementary partners.

Advantages of Python:

• Fast development speed
• Gentle learning curve
• Rich ecosystem
• Suitable for rapid iteration and prototyping

Advantages of Rust:

• Performance close to C/C++
• Memory safety guarantees
• Concurrency-friendly
• Suitable for performance-critical scenarios

Best Practices:

• Use Python to build application skeletons and business logic
• Use Rust to optimize performance bottlenecks
• Seamlessly integrate using tools like PyO3
• Select the appropriate language based on specific scenarios

For developers learning Rust, remember: the value of learning Rust lies not only in mastering a new language but in gaining a new programming mindset. Even if you ultimately do not use Rust extensively in production, it will make you a better programmer.

Do not fear the compiler’s strictness; it is helping you write safer, more reliable code. Every compilation error is a learning opportunity.

References

1. I Used Python for Everything — Until I Discovered Rust: https://medium.com/@kirantechblog/i-used-python-for-everything-until-i-discovered-rust-5aed88c2a78d

Book Recommendations

The book “The Rust Programming Language” (Chinese version, second edition) is an authoritative learning resource written by the Rust core development team and translated by members of the Chinese Rust community. It is suitable for all software developers who wish to evaluate, get started, improve, and research the Rust language, and is considered essential reading for Rust development work.

This book introduces the basic concepts of the Rust language to unique practical tools, covering advanced concepts such as ownership, traits, lifetimes, and safety guarantees, as well as practical tools like pattern matching, error handling, package management, functional features, and concurrency mechanisms. The book includes three complete project development case studies, guiding readers from zero to developing practical Rust projects.

Notably, this book has been updated to the Rust 2021 version, meeting the systematic learning needs of beginners and serving as a reference guide for experienced developers, making it the best entry point for building solid Rust skills.

Recommended Reading

1. Rust: The Performance King Sweeping C/C++/Go?

2. Looking at C++ from the Perspective of a Rust Developer: Revealing Pros and Cons

3. Rust vs Zig: The Battle of Emerging System Programming Languages

4. Essential Design Patterns for Asynchronous Programming in Rust: Enhance Your Code’s Performance and Maintainability