Follow and star to learn new Python skills every day

Due to changes in the public account’s push rules, please click “View” and add “Star” to get exciting technical shares as soon as possible

Source from the internet, please delete if infringing

Python is a scripting language, which has some shortcomings in efficiency and performance compared to compiled languages like C/C++. However, in many cases, the efficiency of Python is not as exaggerated as imagined. This article summarizes some techniques for speeding up Python code execution.

Code Optimization Principles

This article will introduce many techniques for speeding up Python code execution. Before delving into the details of code optimization, it is essential to understand some basic principles of code optimization.

First Basic Principle: Do Not Optimize Prematurely

Many people aim for performance optimization right from the start of coding, “Making the correct program faster is much easier than making a fast program correct.” Therefore, the prerequisite for optimization is that the code works correctly. Premature optimization may overlook the overall performance metrics, and one should not confuse priorities before obtaining global results.

Second Basic Principle: Weigh the Cost of Optimization

Optimization comes at a cost, and it is nearly impossible to solve all performance issues. The usual choice is to trade time for space or space for time. Additionally, the development cost must also be considered.

Third Principle: Do Not Optimize Insignificant Parts

If you optimize every part of the code, these modifications will make the code difficult to read and understand. If your code runs slowly, first identify where the code is slow, which is usually in the inner loops, and focus on optimizing the slow parts. A slight time loss in other areas is usually inconsequential.

Avoid Global Variables

# Not recommended. Code execution time: 26.8 seconds
import math

size = 10000
for x in range(size):
    for y in range(size):
        z = math.sqrt(x) + math.sqrt(y)

Many programmers start by writing simple scripts in Python and often habitually declare global variables, as shown in the code above. However, due to the different implementation of global and local variables, code defined in the global scope runs significantly slower than code defined within functions. By placing script statements into functions, a speed increase of 15% to 30% can typically be achieved.

# Recommended. Code execution time: 20.6 seconds
import math

def main():  # Define in a function to reduce the use of global variables
    size = 10000
    for x in range(size):
        for y in range(size):
            z = math.sqrt(x) + math.sqrt(y)

main()

Avoid

2.1 Avoid Module and Function Attribute Access

# Not recommended. Code execution time: 14.5 seconds
import math

def computeSqrt(size: int):
    result = []
    for i in range(size):
        result.append(math.sqrt(i))
    return result

def main():
    size = 10000
    for _ in range(size):
        result = computeSqrt(size)

main()

Each time you use the <span>.</span> (attribute access operator), it triggers specific methods like <span>__getattribute__()</span> and <span>__getattr__()</span>, which perform dictionary operations, thus incurring additional time overhead. By using the <span>from import</span> statement, attribute access can be eliminated.

# First optimized version. Code execution time: 10.9 seconds
from math import sqrt

def computeSqrt(size: int):
    result = []
    for i in range(size):
        result.append(sqrt(i))  # Avoid using math.sqrt
    return result

def main():
    size = 10000
    for _ in range(size):
        result = computeSqrt(size)

main()

In Section 1, we mentioned that local variable lookup is faster than global variable lookup, so for frequently accessed variables like <span>sqrt</span>, assigning it to a local variable can speed up execution.

# Second optimized version. Code execution time: 9.9 seconds
import math

def computeSqrt(size: int):
    result = []
    sqrt = math.sqrt  # Assign to a local variable
    for i in range(size):
        result.append(sqrt(i))  # Avoid using math.sqrt
    return result

def main():
    size = 10000
    for _ in range(size):
        result = computeSqrt(size)

main()

Besides <span>math.sqrt</span>, there is also the presence of <span>.</span> in the <span>append</span> method of the <span>list</span>. By assigning this method to a local variable, the use of <span>.</span> within the <span>for</span> loop in the <span>computeSqrt</span> function can be completely eliminated.

# Recommended version. Code execution time: 7.9 seconds
import math

def computeSqrt(size: int):
    result = []
    append = result.append
    sqrt = math.sqrt    # Assign to a local variable
    for i in range(size):
        append(sqrt(i))  # Avoid using result.append and math.sqrt
    return result

def main():
    size = 10000
    for _ in range(size):
        result = computeSqrt(size)

main()

2.2 Avoid Class Attribute Access

# Not recommended. Code execution time: 10.4 seconds
import math
from typing import List

class DemoClass:
    def __init__(self, value: int):
        self._value = value
    
    def computeSqrt(self, size: int) -> List[float]:
        result = []
        append = result.append
        sqrt = math.sqrt
        for _ in range(size):
            append(sqrt(self._value))
        return result

def main():
    size = 10000
    for _ in range(size):
        demo_instance = DemoClass(size)
        result = demo_instance.computeSqrt(size)

main()

The principle of avoiding <span>.</span> also applies to class attributes; accessing <span>self._value</span> is slower than accessing a local variable. By assigning frequently accessed class attributes to local variables, the execution speed can be improved.

# Recommended version. Code execution time: 8.0 seconds
import math
from typing import List

class DemoClass:
    def __init__(self, value: int):
        self._value = value
    
    def computeSqrt(self, size: int) -> List[float]:
        result = []
        append = result.append
        sqrt = math.sqrt
        value = self._value
        for _ in range(size):
            append(sqrt(value))  # Avoid using self._value
        return result

def main():
    size = 10000
    for _ in range(size):
        demo_instance = DemoClass(size)
        demo_instance.computeSqrt(size)

main()

Avoid Unnecessary Abstraction

# Not recommended. Code execution time: 0.55 seconds
class DemoClass:
    def __init__(self, value: int):
        self.value = value

    @property
    def value(self) -> int:
        return self._value

    @value.setter
    def value(self, x: int):
        self._value = x

def main():
    size = 1000000
    for i in range(size):
        demo_instance = DemoClass(size)
        value = demo_instance.value
        demo_instance.value = i

main()

Whenever you use an additional processing layer (like decorators, property accessors, or descriptors) to wrap code, it will slow down the code. In most cases, it is necessary to reconsider whether the definition of property accessors is necessary; using <span>getter/setter</span> functions to access properties is often a coding style left over by C/C++ programmers. If it is not necessary, use simple properties instead.

# Recommended version. Code execution time: 0.33 seconds
class DemoClass:
    def __init__(self, value: int):
        self.value = value  # Avoid unnecessary property accessors

def main():
    size = 1000000
    for i in range(size):
        demo_instance = DemoClass(size)
        value = demo_instance.value
        demo_instance.value = i

main()

Avoid Data Copying

4.1 Avoid Meaningless Data Copying

# Not recommended. Code execution time: 6.5 seconds
def main():
    size = 10000
    for _ in range(size):
        value = range(size)
        value_list = [x for x in value]
        square_list = [x * x for x in value_list]

main()

The above code has a completely unnecessary <span>value_list</span>, which creates unnecessary data structures or copies.

# Recommended version. Code execution time: 4.8 seconds
def main():
    size = 10000
    for _ in range(size):
        value = range(size)
        square_list = [x * x for x in value]  # Avoid meaningless copying

main()

Another situation is being overly paranoid about Python’s data sharing mechanism and not understanding or trusting Python’s memory model, leading to the misuse of functions like <span>copy.deepcopy()</span><code><span>. In these codes, copying operations can usually be eliminated.</span>

4.2 Avoid Using Intermediate Variables When Swapping Values

# Not recommended. Code execution time: 0.07 seconds
def main():
    size = 1000000
    for _ in range(size):
        a = 3
        b = 5
        temp = a
        a = b
        b = temp

main()

The above code creates a temporary variable <span>temp</span> when swapping values. Without using an intermediate variable, the code is more concise and runs faster.

# Recommended version. Code execution time: 0.06 seconds
def main():
    size = 1000000
    for _ in range(size):
        a = 3
        b = 5
        a, b = b, a  # Without using an intermediate variable

main()

4.3 Use join for String Concatenation Instead of +

# Not recommended. Code execution time: 2.6 seconds
import string
from typing import List

def concatString(string_list: List[str]) -> str:
    result = ''
    for str_i in string_list:
        result += str_i
    return result

def main():
    string_list = list(string.ascii_letters * 100)
    for _ in range(10000):
        result = concatString(string_list)

main()

When using <span>a + b</span> to concatenate strings, since strings are immutable objects in Python, it allocates a block of memory space, copying <span>a</span> and <span>b</span> into the newly allocated memory space. Therefore, if you want to concatenate <span>n</span> strings, it will produce <span>n-1</span> intermediate results, each requiring memory allocation and copying, severely affecting execution efficiency. Using <span>join()</span> for string concatenation first calculates the total memory space needed, then allocates the required memory all at once and copies each string element into that memory.

# Recommended version. Code execution time: 0.3 seconds
import string
from typing import List

def concatString(string_list: List[str]) -> str:
    return ''.join(string_list)  # Use join instead of +

def main():
    string_list = list(string.ascii_letters * 100)
    for _ in range(10000):
        result = concatString(string_list)

main()

Utilize the Short-Circuiting Feature of if Conditions

# Not recommended. Code execution time: 0.05 seconds
from typing import List

def concatString(string_list: List[str]) -> str:
    abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
    abbr_count = 0
    result = ''
    for str_i in string_list:
        if str_i in abbreviations:
            result += str_i
    return result

def main():
    for _ in range(10000):
        string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
        result = concatString(string_list)

main()

<span>if</span> condition short-circuiting refers to statements like <span>if a and b</span>, where if <span>a</span> is <span>False</span>, it will return directly without evaluating <span>b</span>; for <span>if a or b</span>, if <span>a</span> is <span>True</span>, it will return directly without evaluating <span>b</span>. Therefore, to save execution time, for <span>or</span> statements, the variable with a higher likelihood of being <span>True</span> should be placed before <span>or</span>, while for <span>and</span>, it should be placed later.

# Recommended version. Code execution time: 0.03 seconds
from typing import List

def concatString(string_list: List[str]) -> str:
    abbreviations = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
    abbr_count = 0
    result = ''
    for str_i in string_list:
        if str_i[-1] == '.' and str_i in abbreviations:  # Utilize the short-circuiting feature of if conditions
            result += str_i
    return result

def main():
    for _ in range(10000):
        string_list = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
        result = concatString(string_list)

main()

Loop Optimization

6.1 Use for Loop Instead of while Loop

# Not recommended. Code execution time: 6.7 seconds
def computeSum(size: int) -> int:
    sum_ = 0
    i = 0
    while i < size:
        sum_ += i
        i += 1
    return sum_

def main():
    size = 10000
    for _ in range(size):
        sum_ = computeSum(size)

main()

Python’s <span>for</span> loop is significantly faster than the <span>while</span> loop.

# Recommended version. Code execution time: 4.3 seconds
def computeSum(size: int) -> int:
    sum_ = 0
    for i in range(size):  # Use for loop instead of while loop
        sum_ += i
    return sum_

def main():
    size = 10000
    for _ in range(size):
        sum_ = computeSum(size)

main()

6.2 Use Implicit for Loop Instead of Explicit for Loop

For the above example, we can further replace the explicit <span>for</span> loop with an implicit <span>for</span> loop.

# Recommended version. Code execution time: 1.7 seconds
def computeSum(size: int) -> int:
    return sum(range(size))  # Use implicit for loop instead of explicit for loop

def main():
    size = 10000
    for _ in range(size):
        sum = computeSum(size)

main()

6.3 Reduce Calculations in Inner for Loops

# Not recommended. Code execution time: 12.8 seconds
import math

def main():
    size = 10000
    sqrt = math.sqrt
    for x in range(size):
        for y in range(size):
            z = sqrt(x) + sqrt(y)

main() 

The above code has <span>sqrt(x)</span> located in the inner <span>for</span> loop, which recalculates it every time during the training process, increasing time overhead.

# Recommended version. Code execution time: 7.0 seconds
import math

def main():
    size = 10000
    sqrt = math.sqrt
    for x in range(size):
        sqrt_x = sqrt(x)  # Reduce calculations in inner for loops
        for y in range(size):
            z = sqrt_x + sqrt(y)

main() 

Using numba.jit

We will continue with the previously introduced example and use <span>numba.jit</span> on top of it. <span>numba</span> can JIT compile Python functions into machine code for execution, greatly improving code execution speed. For more information about <span>numba</span>, see the homepage below:

http://numba.pydata.org/numba.pydata.org

# Recommended version. Code execution time: 0.62 seconds
import numba

@numba.jit
def computeSum(size: float) -> int:
    sum = 0
    for i in range(size):
        sum += i
    return sum

def main():
    size = 10000
    for _ in range(size):
        sum = computeSum(size)

main()

Select Appropriate Data Structures

Python’s built-in data structures such as <span>str</span>, <span>tuple</span>, <span>list</span>, <span>set</span>, and <span>dict</span> are all implemented in C, making them very fast. It is nearly impossible to achieve the performance of built-in data structures by implementing new ones yourself.

<span>list</span> is similar to C++’s <span>std::vector</span>, which is a dynamic array. It pre-allocates a certain amount of memory space, and when the pre-allocated memory space is exhausted and more elements are added, it requests a larger block of memory and copies all existing elements over, then destroys the previous memory space before inserting new elements. When deleting elements, the operation is similar; when the used memory space is less than half of the pre-allocated memory space, it requests a smaller block of memory, performs a copy of the elements, and then destroys the original larger memory space.

Therefore, if there are frequent additions and deletions, and the number of elements added or deleted is large, the efficiency of <span>list</span> is not high. In this case, consider using <span>collections.deque</span>. <span>collections.deque</span> is a double-ended queue that has the characteristics of both stacks and queues, allowing for <span>O(1)</span> complexity insertions and deletions at both ends.

<span>list</span> also has very slow lookup operations. When frequent lookups of certain elements in a <span>list</span> are needed, or when there is frequent ordered access to these elements, you can use <span>bisect</span> to maintain the order of the <span>list</span> object and perform binary searches to improve lookup efficiency.

Another common requirement is to find the minimum or maximum value; in this case, you can use the <span>heapq</span> module to convert a <span>list</span> into a heap, making the time complexity for obtaining the minimum value <span>O(1)</span>.

Click to follow the public account below to get free Python public courses and hundreds of gigabytes of learning materials organized by experts, including but not limited to Python e-books, tutorials, project orders, source code, etc.

▲ Click to follow - Free to receive

Recommended Reading
Python course design based on Django + vue implementation of sports mall system source code + database + detailed project documentation
Everything is under control: this Python script automatically reports progress to you
PyCharm + Docker: Create the most comfortable deep learning furnace
Use Python to draw a beautiful rabbit source code + detailed code comments, Python draw rabbit year mascot source code, based on turtle implementation


Click to read the original text