With each release of Python, new modules are added and better ways of doing things are introduced. While we have all grown accustomed to using good old Python libraries and certain methods, it is time to upgrade and take advantage of the new and improved modules and their features.
Use Pathlib Instead of OS
Pathlib is definitely one of the biggest additions to the Python standard library in recent times. It has been part of the standard library since Python 3.4, yet many people still use the os module for filesystem operations.
However, pathlib has many advantages over the old os.path – while the os module represents paths in a raw string format, pathlib uses an object-oriented style, making it more readable and natural to write:
from pathlib import Path
import os.path # Old way
two_dirs_up = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) # New way, more readable
two_dirs_up = Path(__file__).resolve().parent.parentThe fact that paths are treated as objects rather than strings also allows you to create an object once and then look up its properties or manipulate it:
readme = Path("README.md").resolve()
print(f"Absolute path: {readme.absolute()}") # Absolute path: /home/martin/some/path/README.md
print(f"File name: {readme.name}") # File name: README.md
print(f"Path root: {readme.root}") # Path root: /
print(f"Parent directory: {readme.parent}") # Parent directory: /home/martin/some/path
print(f"File extension: {readme.suffix}") # File extension: .md
print(f"Is it absolute: {readme.is_absolute()}") # Is it absolute: TrueOne of my favorite features of pathlib is the ability to concatenate paths using the / (“division”) operator:
# Operators:
etc = Path('/etc')
joined = etc / "cron.d" / "anacron"
print(f"Exists? - {joined.exists()}") # Exists? - TrueIt is important to note that pathlib is just a replacement for os.path and not the entire os module; it also includes functionality from the glob module, so if you are accustomed to using os.path with glob.glob, you can fully replace them with pathlib.
In the snippets above, we showcased some convenient path operations and object properties, but pathlib also includes all the methods you are used to from os.path, such as:
print(f"Working directory: {Path.cwd()}") # same as os.getcwd()
# Working directory: /home/martin/some/path
Path.mkdir(Path.cwd() / "new_dir", exist_ok=True) # same as os.makedirs()
print(Path("README.md").resolve()) # same as os.path.abspath()
# /home/martin/some/path/README.md
print(Path.home()) # same as os.path.expanduser()
# /home/martinFor a complete mapping of os.path functions to new functions in pathlib, refer to the official documentation.
Use Secrets Instead of OS
Speaking of the os module, another part you should stop using is os.urandom. Instead, you should use the new secrets module available since Python 3.6:
# Old way:
import os
length = 64
value = os.urandom(length)
print(f"Bytes: {value}") # Bytes: b'\xfa\xf3...\xf2\x1b\xf5\xb6'
print(f"Hex: {value.hex()}") # Hex: faf3cc656370e31a938e7...33d9b023c3c24f1bf5
# New way:
import secrets
value = secrets.token_bytes(length)
print(f"Bytes: {value}") # Bytes: b'U\xe9n\x87...\x85>\x04j:\xb0'
value = secrets.token_hex(length)
print(f"Hex: {value}") # Hex: fb5dd85e7d73f7a08b8e3...4fd9f95beb08d77391Using os.urandom is not the issue here; the reason the secrets module was introduced is that people were using the random module to generate passwords, etc., even though the random module does not produce cryptographically secure tokens.
According to the documentation, the random module should not be used for security purposes. You should use secrets or os.urandom, but the secrets module is definitely preferable as it is newer and contains some utility/convenience methods for hex tokens and URL-safe tokens.
Use Zoneinfo Instead of Pytz
Before Python 3.9, there was no built-in library for timezone operations, so everyone was using pytz, but now we have zoneinfo in the standard library, so it is time to switch.
from datetime import datetime
import pytz # pip install pytz
dt = datetime(2022, 6, 4)
nyc = pytz.timezone("America/New_York")
localized = nyc.localize(dt)
print(f"Datetime: {localized}, Timezone: {localized.tzname()}, TZ Info: {localized.tzinfo}") # New way:
from zoneinfo import ZoneInfo
nyc = ZoneInfo("America/New_York")
localized = datetime(2022, 6, 4, tzinfo=nyc)
print(f"Datetime: {localized}, Timezone: {localized.tzname()}, TZ Info: {localized.tzinfo}") # Datetime: 2022-06-04 00:00:00-04:00, Timezone: EDT, TZ Info: America/New_YorkThe datetime module delegates all timezone operations to the abstract base class datetime.tzinfo, which requires a concrete implementation – before this module, which likely came from pytz, was introduced. Now we have zoneinfo in the standard library that we can use.
However, there is a caveat when using zoneinfo – it assumes that timezone data is available on the system, which is the case for UNIX systems. If your system does not have timezone data, then you should use the tzdata package, which is a first-party library maintained by the CPython core developers containing the IANA timezone database.
Dataclasses
A significant addition in Python 3.7 is the dataclasses package, which is an alternative to namedtuple.
You might wonder why there is a need to replace namedtuple? Here are some reasons you should consider switching to dataclasses:
- 1. They can be mutable
- 2. They provide default repr, eq, init, and hash magic methods
- 3. They allow specifying default values
- 4. They support inheritance. Additionally, dataclasses also support frozen and slots (starting from 3.10) attributes to provide parity with the features of namedtuple.
The switch should not be too difficult as you only need to change the definition:
# Old way:
# from collections import namedtuple
from typing import NamedTuple
import sys
User = NamedTuple("User", [("name", str), ("surname", str), ("password", bytes)])
u = User("John", "Doe", b'tfeL+uD...\xd2')
print(f"Size: {sys.getsizeof(u)}") # Size: 64
# New way:
from dataclasses import dataclass
@dataclass()
class User:
name: str
surname: str
password: bytes
u = User("John", "Doe", b'tfeL+uD...\xd2')
print(u) # User(name='John', surname='Doe', password=b'tfeL+uD...\xd2')
print(f"Size: {sys.getsizeof(u)}, {sys.getsizeof(u) + sys.getsizeof(vars(u))}") # Size: 48, 152In the above code, we also included a size comparison because this is one of the larger differences between namedtuple and dataclasses. As seen, namedtuple is significantly smaller in size, which is due to dataclasses using dict to represent attributes.
As for speed comparison, unless you plan to create millions of instances, the access time for attributes should be roughly the same, or not significant:
import timeit
setup = '''from typing import NamedTuple
User = NamedTuple("User", [("name", str), ("surname", str), ("password", bytes)])
u = User("John", "Doe", b'')'''
print(f"Access speed: {min(timeit.repeat('u.name', setup=setup, number=10000000))}") # Access speed: 0.16838401100540068
setup = '''from dataclasses import dataclass
@dataclass(slots=True)
class User:
name: str
surname: str
password: bytes
u = User("John", "Doe", b'')'''
print(f"Access speed: {min(timeit.repeat('u.name', setup=setup, number=10000000))}") # Access speed: 0.17728697300481144If the above has convinced you to switch to dataclasses, give it a try soon.
On the other hand, if you do not want to switch and for some reason really want to use namedtuple, then you should at least use the typing module instead of NamedTuple from collections:
# Bad way:
from collections import namedtuple
Point = namedtuple("Point", ["x", "y"])
# Better way:
from typing import NamedTuple
class Point(NamedTuple):
x: float
y: floatFinally, if you are using neither namedtuple nor dataclasses, you might want to consider using Pydantic directly.
Proper Logging Instead of Print
This is not the latest addition to the standard library, but it is worth using – you should use proper logging instead of print statements. If you are debugging issues locally, you can use print, but for any production-ready program that runs without user intervention, proper logging is a must.
Especially considering that setting up Python logging is very simple:
import logging
logging.basicConfig(
filename='application.log',
level=logging.WARNING,
format='[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s',
datefmt='%H:%M:%S')
logging.error("Some serious error occurred.") # [12:52:35] {:1} ERROR - Some serious error occurred.
logging.warning('Some warning.') # [12:52:35] {:1} WARNING - Some warning.The simple configuration above will provide you with an excellent debugging experience compared to print statements. Most importantly, you can further customize the logging library to log to different locations, change log levels, automatically rotate logs, etc.
Use f-strings Instead of Format
Python includes many ways to format strings, including C-style formatting, f-strings, template strings, or the .format function. However, one of them – f-strings – formats string literals, which are more natural to write, more readable, and the fastest among the mentioned options.
Therefore, I think there is no need to argue or explain why to use them; however, there are certain situations where f-strings cannot be used:
The only reason to use % format is for logging:
import logging
things = "something happened..."
logger = logging.getLogger(__name__)
logger.error("Message: %s", things) # Evaluates inside the logger method
logger.error(f"Message: {things}") # Immediately evaluatedIn the example above, if you use f-strings, the expressions will be evaluated immediately, while with C-style formatting, the substitutions will be delayed until they are actually needed, which is important for message grouping, where all messages with the same template can be logged as one. This does not apply to f-strings because the template is filled with data before being passed to the logger.
Additionally, there are some things that f-strings cannot do at all, such as filling a template at runtime – i.e., dynamic formatting – which is why f-strings are referred to as literal string formatting:
# Dynamically setting templates and their parameters
def func(tpl: str, param1: str, param2: str) -> str:
return tpl.format(param=param1, param2=param2)
some_template = "First template: {param1}, {param2}"
another_template = "Other template: {param1} and {param2}"
print(func(some_template, "Hello", "World"))
print(func(another_template, "Hello", "Python")) # Dynamically reusing the same template with different parameters.
inputs = ["Hello", "World", "!"]
template = "Here's some dynamic value: {value}"
for value in inputs:
print(template.format(value=value))Most importantly, use f-strings whenever possible because they are more readable and higher performance, but note that in some cases, other formatting styles are still preferred and/or required.
Use Tomllib Instead of Tomli
TOML is a widely used configuration format, especially important for Python’s tools and ecosystem as it is used for pyproject.toml configuration files. Until now, you had to use an external library to manage TOML files, but starting from Python 3.11, there will be a built-in library called tomllib based on the toml package.
So, once you switch to Python 3.11, you should get into the habit of using import tomllib instead of import tomli. One less dependency to worry about!
# import tomli as tomllib
import tomllib
with open("pyproject.toml", "rb") as f:
config = tomllib.load(f)
print(config) # {'project': {'authors': [{'email': '[email protected]',
# 'name': 'Martin Heinz'}],
# 'dependencies': ['flask', 'requests'],
# 'description': 'Example Package',
# 'name': 'some-app',
# 'version': '0.1.0'}}
toml_string = """[project]
name = "another-app"
description = "Example Package"
version = "0.1.1"""
config = tomllib.loads(toml_string)
print(config) # {'project': {'name': 'another-app', 'description': 'Example Package', 'version': '0.1.1'}}Use Setuptools Instead of Distutils
The last one is more of a deprecation notice:
Since Distutils has been deprecated, similarly, any functions or objects from distutils are also discouraged. Setuptools is intended to replace or deprecate all such uses.
It is time to say goodbye to the distutils package and switch to setuptools. The setuptools documentation provides guidance on how to replace distutils usage, and besides that, PEP 632 also provides migration suggestions for parts of distutils that are not covered by setuptools.
Conclusion
Every new version of Python brings new features, so I recommend checking out the “New Modules”, “Deprecated Modules”, and “Removed Modules” sections in the Python release notes, which is a great way to stay informed about major changes in the Python standard library. This way, you can continuously integrate new features and best practices into your projects.
– EOF –
Reference link: Click the bottom left corner to read the original text. For academic sharing only, if there is any infringement, please delete immediately.
Editor / Garvey
Reviewer / Fan Ruiqiang
Verification / Fan Ruiqiang
Click below
Follow us