Curated List of 100 Essential Python Libraries

Why is Python so popular and why are so many people learning it? It’s simply because it’s easy to learn, powerful, and the community is very active with plenty of resources. This language covers various aspects, such as automation testing, operations, web scraping, data analysis, machine learning, finance, backend development, cloud computing, and game development.

A skyscraper rises from the ground; the reason Python is so powerful is due to the support of countless libraries. Today, we will organize and overview the entire treasure trove of Python.

I have roughly listed the applications of Python libraries, and there are about 20 different directions. The left side is responsible for offense (external), and each item is a boutique, a flagship product in Python applications; the right side is responsible for defense (internal), involving lower-level configurations, which form the foundation of the entire building.

Next, we will select boutique libraries from each field and present them to everyone. After reading, you will definitely exclaim, ‘Wow, I’ve never seen so many libraries before!’

1. The Infrastructure of the Python Skyscraper

Environment Management

Tools for managing Python versions and environments.

pyenv: A simple Python version management tool.

Vex: Executes commands in a virtual environment.

virtualenv: A tool for creating isolated Python environments.

buildout: Uses declarative configuration management after initializing in an isolated environment.

Package Management

Tools for managing packages and dependencies.

pip: Python package and dependency management tool.

pip-tools: A set of tools to ensure Python package dependencies are updated.

pipenv: The new generation package management tool officially recommended by Python.

poetry: A package management tool that can completely replace setup.py.

conda: A cross-platform Python binary package management tool.

Curdling: A command-line tool for managing Python packages.

wheel: The new standard for Python distribution, intended to replace eggs.

Distribution

Packages into executable files for distribution.

PyInstaller: Converts Python programs into standalone executable files (cross-platform).

cx_Freeze: Converts Python programs into executable files with a dynamic link library.

dh-virtualenv: Builds and publishes a virtualenv virtual environment as a Debian package.

Nuitka: Compiles scripts, modules, and packages into executable files or extension modules.

py2app: Converts Python scripts into standalone packages (Mac OS X).

py2exe: Converts Python scripts into standalone packages (Windows).

pynsist: A tool for creating Windows installers that can package Python itself.

Configuration

Libraries for saving and parsing configurations.

config: A hierarchical configuration module written by the author of the logging module.

ConfigObj: An INI file parser with validation capabilities.

ConfigParser: (Python standard library) INI file parser.

profig: Configuration through various formats with numerical conversion functionality.

python-decouple: Completely isolates settings from code.

2. Management Configuration of the Python Skyscraper

Files

aiofiles: Provides asynchronous file operations based on asyncio.

imghdr: (Python standard library) Detects image types.

mimetypes: (Python standard library) Maps filenames to MIME types.

path.py: A module that encapsulates os.path.

pathlib: (Python 3.4+ standard library) A cross-platform, object-oriented path manipulation library.

python-magic: A Python interface for the third-party library libmagic for file type detection.

Unipath: Object-oriented file and directory manipulation.

watchdog: API and shell tools for managing file system events.

pofile: A file processing tool for beginners.

Date and Time

Libraries for manipulating dates and times.

arrow: A better Python date and time manipulation library.

Chronyk: A library for Python 3 that parses handwritten date and time formats.

dateutil: An extension of the Python datetime module.

delorean: A library that solves tricky date handling issues in Python.

maya: A user-friendly time manipulation library.

moment: A Python library for handling time and dates, inspired by Moment.js.

pendulum: A time manipulation library with more explicit and predictable behavior than arrow.

PyTime: A simple and easy-to-use Python module for manipulating dates/times via strings.

pytz: Modern and historical versions of world timezone definitions, bringing timezone databases into Python.

when.py: Provides user-friendly functions to assist users with common date and time operations.

potime: A code tool for measuring execution time.

Text Processing

Libraries for parsing and manipulating text.

• General

chardet: Character encoding detector, compatible with Python2 and Python3.

difflib: (Python standard library) Helps us perform differential comparisons.

ftfy: Makes Unicode text more complete and coherent.

fuzzywuzzy: Fuzzy string matching.

Levenshtein: Quickly calculates edit distances and string similarities.

pangu.py: Adds spaces between Chinese, Japanese, Korean characters and numbers/letters.

pypinyin: A tool for converting Chinese characters to Pinyin in Python.

shortuuid: A generator library for generating concise, understandable, URL-safe UUIDs.

simplejson: JSON encoder/decoder for Python.

unidecode: ASCII transliterations of Unicode text.

uniout: Prints readable characters instead of escaped strings.

xpinyin: A library for converting Chinese characters to Pinyin.

yfiglet-figlet: Python implementation of pyfiglet -figlet.

flashtext: An efficient text search and replace library.

• Slugification

awesome-slugify: A Python slugification library that retains Unicode.

python-slugify: A Python slugification library that converts Unicode to ASCII.

unicode-slugify: A slug tool that generates Unicode slugs, requires Django.

• Parsers

phonenumbers: Parses, formats, stores, and validates phone numbers.

PLY: Python implementation of lex and yacc parsing tools.

Pygments: A general syntax highlighting tool.

pyparsing: A framework for generating general parsers.

python-nameparser: Breaks down a person’s name into separate components.

python-user-agents: Browser user agent parser.

sqlparse: An unvalidated SQL parser.

Office Document Format Processing

• General

tablib: A module for processing tabular data.

⭐python-office: A third-party library for automating office tasks.

• Office

Marmir: Converts input Python data structures to electronic forms.

openpyxl: A library for reading/writing Excel 2010 xlsx/xlsm/xltx/xltm files.

pyexcel: A library providing a unified API for reading, writing, and manipulating Excel files.

python-docx: Reads, queries, and modifies Microsoft Word 2007/2008 docx files.

relatorio: Templates for OpenDocument files.

unoconv: Converts between any file formats supported by LibreOffice/OpenOffice.

XlsxWriter: A Python module for creating Excel .xlsx files.

xlwings: A library that makes it easy to call Python from Excel (and vice versa), based on the BSD license.

xlwt: Reads and writes data and formatting information in Excel files.

poexcel: Processes Excel.

• PDF

PDFMiner: A tool for extracting information from PDF documents.

popdf: A tool for handling PDFs.

PyPDF2: A library for splitting, merging, and converting PDF pages.

ReportLab: Quickly creates rich text PDF documents.

• Markdown

Mistune: A fast and fully featured pure Python Markdown parser.

Python-Markdown: The Python implementation of John Gruber’s Markdown.

Python-Markdown2: A pure Python Markdown parser that is faster and more accurate than Python-Markdown.

• YAML

PyYAML: The Python version of the YAML parser.

• CSV

csvkit: A tool for converting and manipulating CSV.

Configuration

Libraries for saving and parsing configurations.

config: A hierarchical configuration module written by the author of the logging module.

ConfigObj: An INI file parser with validation capabilities.

ConfigParser: (Python standard library) INI file parser.

profig: Configuration through various formats with numerical conversion functionality.

python-decouple: Completely isolates settings from code.

Command Line Tools

Libraries for creating command line programs.

• Command Line Program Development

asciimatics: A cross-platform, full-screen terminal package (i.e., mouse/keyboard input and colored, positioned text output), with a complete API for complex animations and special effects.

cement: A command line program framework for Python.

click: A package for creating beautiful command line interfaces through composition.

cliff: A framework for creating command line programs that can create multi-layered command line applications.

clint: A command line program tool for Python.

colorama: Cross-platform colored terminal text.

docopt: A Python-style command line argument parser.

Gooey: A command that turns command line programs into GUI applications.

python-prompt-toolkit: A library for building powerful interactive command line programs.

python-fire: A library from Google for building command line interfaces based on Python classes.

Pythonpy: Directly executes any Python instruction in the command line.

• Productivity Tools

aws-cli: A universal command line interface for Amazon Web Services.

bashplotlib: Basic plotting in the terminal.

caniusepython3: Determines which projects hinder your transition to Python 3.

cookiecutter: A command line tool for creating projects from cookiecutters (project templates).

doitlive: A tool for live demonstrations in the terminal.

pyftpdlib: A fast and scalable Python FTP service library.

howdoi: Get instant programming answers via the command line.

httpie: A command line HTTP client, a better alternative to cURL, with improved usability.

PathPicker: Selects files from bash output.

percol: Adds interactive selection to the traditional UNIX shell pipeline concept.

SAWS: An enhanced version of the AWS command line.

thefuck: Corrects your previous command line instructions.

mycli: A MySQL command line client with autocomplete and syntax highlighting features.

pgcli: A Postgres command line tool with autocomplete and syntax highlighting features.

try: A never simpler command line tool for trying out Python libraries.

3. The Warehouse of the Python Skyscraper

Databases

Databases implemented in Python.

pickleDB: A simple, lightweight key-value storage database.

PipelineDB: A streaming SQL database.

TinyDB: A micro, document-oriented database.

ZODB: A Python native object database, a key-value and object graph database.

Database Drivers

Libraries for connecting and operating databases.

• MySQL: awesome-mysql series

aiomysql: An asynchronous MySQL database operation library based on asyncio.

mysql-python: A MySQL database connector for Python.

mysqlclient: A branch of mysql-python that supports Python 3.

oursql: A better MySQL connector that supports native prepared statements and BLOBs.

PyMySQL: A pure Python MySQL driver, compatible with mysql-python.

• PostgreSQL

psycopg2: The most popular PostgreSQL adapter in Python.

queries: A wrapper for the psycopg2 library to interact with PostgreSQL.

txpostgres: An asynchronous PostgreSQL driver based on Twisted.

• Other Relational Databases

apsw: Another Python SQLite wrapper.

dataset: Stores Python dictionaries in databases.

pymssql: A simple Microsoft SQL Server database interface.

• NoSQL Databases

asyncio-redis: An asyncio-based Redis client (PEP 3156).

cassandra-python-driver: Python driver for Cassandra.

HappyBase: A developer-friendly library designed for Apache HBase.

Plyvel: A fast and feature-rich Python interface for LevelDB.

py2neo: A Python client for the Neo4j restful interface.

pycassa: Python Thrift driver for Cassandra.

PyMongo: The official Python client for MongoDB.

redis-py: The Python client for Redis.

telephus: A Twisted-based Cassandra client.

txRedis: A Twisted-based Redis client.

4. Network Communication of the Python Skyscraper

Networking

Libraries for network programming.

asyncio: (Python standard library) Asynchronous I/O, event loop, coroutines, and tasks.

Twisted: An event-driven networking engine.

pulsar: An event-driven concurrency framework.

diesel: An event I/O framework based on Greenlet.

pyzmq: A Python wrapper for the ZeroMQ messaging library.

Toapi: A lightweight, simple, fast Flask library dedicated to providing API services for all websites.

txZMQ: A Python wrapper for the ZeroMQ messaging library based on Twisted.

HTTP

Libraries for using HTTP.

aiohttp: An asynchronous HTTP networking library based on asyncio.

requests: A user-friendly HTTP request library.

grequests: requests library + gevent, for asynchronous HTTP requests.

httplib2: A comprehensive HTTP client library.

treq: A Python API similar to requests built on top of the Twisted HTTP client.

urllib3: A thread-safe connection pool with file post support, a clear and friendly HTTP library.

WebSocket: Libraries for using WebSocket.

AutobahnPython: WebSocket & WAMP for Python based on Twisted and asyncio.

Crossbar: Open-source unified application routing (WebSocket & WAMP for Python on Autobahn).

django-socketio: WebSockets for Django.

WebSocket-for-Python: A WebSocket client and server library for Python2/3 and PyPy.

WSGI Servers

WSGI-compatible web servers.

gunicorn: Pre-forked, partly written in C.

uwsgi: The goal of the uwsgi project is to develop a suite of full-stack tools for building hosting services, written in C.

bjoern: Asynchronous, very fast, written in C.

fapws3: Asynchronous (only for the network side), written in C.

meinheld: Asynchronous, partly written in C.

netius: Asynchronous, very fast.

paste: Multi-threaded, stable, well-tested.

rocket: Multi-threaded.

waitress: Multi-threaded, powering the Pyramid framework.

Werkzeug: A WSGI utility library powering Flask, and can be easily embedded into your project.

Web Frameworks

Full-stack web frameworks.

Django: The most popular web framework in the Python community.

Flask: A micro framework for Python.

pyramid: A small, fast, and grounded open-source Python web framework.

Bottle: A fast, small, lightweight WSGI micro web framework.

CherryPy: An ultra-simple Python web framework that adheres to the HTTP/1.1 protocol and has a WSGI thread pool.

TurboGears: A micro framework that can be extended into a full-stack solution.

Tornado: A web framework and asynchronous networking library.

sanic: An asynchronous networking framework based on Python 3.5+.

starlette: A lightweight, high-performance ASGI framework.

E-Commerce

Frameworks and libraries for e-commerce and payments.

django-oscar: An open-source e-commerce framework for Django.

django-shop: A shop system based on Django.

Cartridge: A shopping cart application built on Mezzanine.

shoop: An open-source e-commerce platform based on Django.

alipay: An unofficial Python API for Alipay.

merchant: A Django application that can receive payments from various payment platforms.

money: A currency library with optional CLDR backend localization formats, providing extensible currency conversion solutions.

python-currencies: Displays currency formats and their values.

RESTful API

Libraries for developing RESTful APIs.

• Django

django-rest-framework: A powerful and flexible toolkit for building web APIs.

django-tastypie: Develop APIs for Django applications.

django-formapi: Create JSON APIs for Django form validation.

• Flask

flask-api: Browsable Web APIs developed for Flask.

flask-restful: Quickly create REST APIs for Flask.

flask-restless: Create RESTful APIs for SQLAlchemy-defined database models.

flask-api-utils: Handle API representation and validation for Flask.

eve: A REST API framework powered by Flask, MongoDB, etc.

• Pyramid

cornice: A REST framework for Pyramid.

• Framework-independent

falcon: A high-performance framework for building cloud APIs and web app backends.

sandman: Automatically creates REST APIs for existing database-driven systems.

restless: A framework-independent REST framework, based on lessons learned from Tastypie.

ripozo: Quickly creates REST/HATEOAS/Hypermedia APIs.

Template Engines

Libraries and tools for template generation and lexical parsing.

Jinja2: A modern, designer-friendly template engine.

Chameleon: An HTML/XML template engine that mimics ZPT (Zope Page Templates) and is optimized for speed.

Genshi: A Python template tool for generating web-aware results.

Mako: A high-speed lightweight template for the Python platform.

Queues

Libraries for processing events and task queues.

celery: An asynchronous task queue/job queue based on distributed messaging.

huey: A small multi-threaded task queue.

mrq: Mr. Queue – a distributed worker task queue for Python, using Redis and gevent.

rq: A simple Python job queue.

simpleq: A simple, infinitely scalable queue based on Amazon SQS.

Search

Libraries and software for indexing data and executing search queries.

django-haystack: Modular search for Django.

elasticsearch-py: The official low-level Python client for Elasticsearch.

elasticsearch-dsl-py: The official high-level Python client for Elasticsearch.

solrpy: A Python client for Solr.

Whoosh: A fast pure Python search engine library.

Email

Libraries for sending and parsing emails.

django-celery-ses: Django email backend with AWS SES and Celery.

envelopes: An email library for human use.

flanker: An email address and MIME parsing library.

imbox: A Python IMAP library.

inbox.py: A Python SMTP server.

inbox: An open-source email toolbox.

lamson: A Python-style SMTP application server.

mailjet: Implementation of the Mailjet API for bulk email sending, statistics, etc.

marrow.mailer: A high-performance, extensible email dispatch framework.

modoboa: An email hosting and management platform with a modern, minimalist web UI.

pyzmail: Create, send, and parse emails.

Talon: Mailgun library for extracting information and signatures.

yagmail: A GMAIL/SMTP client designed to make sending emails as simple as possible.

URL Handling

Libraries for parsing URLs.

furl: A small Python library that makes URL handling easier.

purl: A simple, immutable URL class with a clean API for querying and processing.

pyshorteners: A pure Python URL shortening library.

shorturl: Generates short URLs and similar to bit.ly short links in Python.

webargs: A library for parsing HTTP request parameters, with built-in support for popular web frameworks including Flask, Django, Bottle, Tornado, and Pyramid.

5. The Power System of the Python Skyscraper

Processes

Libraries for operating system process initiation and communication.

envoy: More user-friendly than the Python subprocess module.

sarge: Another encapsulation of the subprocess module.

sh: A complete subprocess replacement library.

Concurrency and Parallelism

Libraries for performing concurrent and parallel operations.

multiprocessing: (Python standard library) Process-based ‘thread’ interface.

threading: (Python standard library) A higher-level thread interface.

eventlet: An asynchronous framework that supports WSGI.

gevent: A Python networking library based on coroutines, using greenlet.

Tomorrow: A magical decorator syntax implementation for generating asynchronous code.

uvloop: A super-fast implementation of the asyncio event loop on top of libuv.

It’s astonishing how many libraries there are in Python. The above 100+ libraries are just a part of the Python Empire skyscraper, responsible for constructing the internal foundation of the building. The editor will continue to introduce a comprehensive list of Python application libraries, so stay tuned.