Why Our Company Still Uses Python for Development

In recent years, we have often seen some large companies that heavily used Python migrate to other language tech stacks. But what about small companies or small teams?

I have always wanted to understand how companies that still insist on using Python, and have a certain scale of business, use the Python tech stack for development, what difficulties or lessons they encounter, and what excellent experiences they have?

By chance, I saw an answer under the question “**Why do software companies rarely use Python for web development?**” on Zhihu, and I would like to share it with everyone.

Response:

I have been using Python for over 10 years, and the longest project I maintain has an annual transaction volume of several hundred million, which is an e-commerce platform. The concurrency is not large, usually dozens of concurrent requests, and during holidays it may go over 100. At its peak, I have not seen it exceed 200. The total number of orders in the database is about 50 million, increasing by tens of thousands daily. The project has been running for seven or eight years, still using Python 2.7 + Django 1.8, and there are no plans to upgrade.

Currently, we have one 4-core 8G server and three 8-core 16G servers on Alibaba Cloud, with the database and Redis also on Alibaba Cloud. The annual cost is about less than 50,000, and we use Qiniu for CDN, which costs about several tens of thousands each year. There are three programmers, including myself, maintaining it, and we basically add new features every week. After several years of adjustments, the effective code is estimated to be less than 70%, and some code is not in use due to business reasons.

In 2021, we developed another system using Python 3.8 + Django 3. There is usually not much volume, but it spikes during holidays. So far, the highest record is 350 orders per minute and 150,000 orders in a day, with a transaction volume of about 15 million that day. We usually have two 8-core 16G servers configured, and during holidays, we scale up to six servers, and the database is temporarily upgraded as well. There are four programmers, including myself, maintaining it.

Additionally, we have several smaller projects that haven’t really taken off, with not much volume. Typically, two people are responsible for one project, with one person simultaneously managing two projects in a cross-functional manner.

Currently, the entire company’s backend tech stack is Python + Django + Gunicorn (with a small project using Tornado). The company has accumulated some basic frameworks based on Django, and the entire company is not large, with about 14 or 15 programmers who are all familiar with this framework. Newcomers usually go through a process of strengthening their Python basics -> learning Django -> learning the company’s framework -> entering project development.

The company has higher requirements for naming, style, etc., and pays more attention during code reviews. After becoming familiar, everyone basically has a good tacit understanding, so the disadvantages of Python as a dynamic language have not been reflected much.

In the early days, due to lack of experience, we encountered situations where concurrent requests caused crashes. Later, we upgraded the database (initially self-built) and implemented some Redis caching, which has reduced these occurrences significantly.

Some query languages constructed by Django are overly complex or not optimized, leading to slow queries. The current solution is to regularly monitor slow logs, find the problematic code for optimization, and the database itself needs to be upgraded according to business needs. This is true for any language.

I have been exposed to most programming languages, but I find that Python allows me to express my intentions to the computer as easily as my mother tongue.

From my perspective, once a programmer is familiar with Python, they only need to understand the business and convert requirements into code, without spending too much time on technicalities. Python has a rich library, and most problems encountered have ready-made solutions. Django’s ORM is also excellent, allowing programmers to easily manipulate the database without worrying about table structure changes or complex queries.

There are downsides, such as Python being somewhat cumbersome, especially as projects grow larger and more complex, leading to longer startup and loading times, and increased memory usage once running.

While Django’s ORM brings convenience, it also leads to some inefficient code. For example, some people construct overly complex queries, resulting in too many joined tables and long query times, or they often fetch unnecessary fields all at once, and perform numerous data queries within for loops.

However, I believe these downsides are not fatal, because compared to labor costs and development efficiency, the cost of increasing cloud server resources is very low. Moreover, for performance, most projects do not reach the stage that requires optimization before being abandoned. Some norms or usage methods can be improved through training, and the quality of code written by most people will gradually improve.

In addition to web development, we also use Python for some hardware devices (most of which are Linux-based single-board computers, like Raspberry Pi, 7688, etc.). The advantage is that we can develop on a computer and run it directly on the device without needing to hire embedded engineers; once the hardware call parts are encapsulated, any backend developer in the company can develop.

We also use Python for image processing, crawling, automated testing, and CICD.

For small teams, Python’s low threshold and high efficiency are more valuable than the elusive type of performance loss, provided that norms are established, quality is emphasized, and continuous attention and optimization are maintained.

I didn’t expect this answer to attract so much attention, so I will add a few more points.

The system mentioned earlier, which processes 350 orders per minute, mainly aggregates orders from several takeaway platforms into the system, allowing merchants to use the aggregated delivery platforms to call riders for delivery, involving synchronization of takeaway orders and delivery orders, as well as some management functions.

Order notifications from takeaway platforms (new orders, order status changes, etc.) are sent to our system through HTTP requests. In the early days, we used a synchronous method, which meant that upon receiving a request, we would call the takeaway platform’s order query interface (and several other supporting interfaces) to obtain order detail data, create an order in our database, and then respond. This required a significant amount of time due to numerous network requests, and as soon as the concurrency increased slightly, we couldn’t handle it. We tried opening several machines and processes, but the effect was minimal.

I remember that we could only handle about 30 orders per minute at our peak, and any more led to noticeable slow responses, while the takeaway platforms required us to respond within a specified time. Therefore, this synchronous processing method could not be sustained for long before we encountered significant bottlenecks. We also tried multi-threaded task queues, but the results were unsatisfactory and there was a risk of task loss.

Later, we used Celery; upon receiving a notification, we would place the message into the Celery queue and return, allowing Celery worker processes to handle it slowly, thus avoiding being overwhelmed during peak times. Since placing messages into the Celery queue is an extremely fast operation, the system can respond immediately to notifications from the takeaway platform.

Based on the backlog of messages, we adjust the number of Celery worker processes appropriately and can allocate different queues based on message priority to ensure that new order notifications are processed in a timely manner, allowing merchants to know about new orders needing processing as soon as possible.

Initially, we used Redis for Celery’s message distribution, but later switched to RabbitMQ for easier monitoring. After several years of iteration, we are relatively confident in coping with peak times during holidays; we temporarily increase cloud resources as needed, and Celery worker processes are set to auto-scale. In principle, unless we encounter extremely extreme situations, we are confident that we can handle it.

In addition to the aforementioned projects, about seven or eight years ago, we used Python 2 + Django 1.8 to develop a data reporting system for the government. Each year, we would open a week for enterprises to fill in data, with about 4,000 to 5,000 enterprises participating, each filling out seven or eight forms, and the concurrency was not monitored at the time but conservatively estimated to be in the dozens.

Initially, we ran it using Django’s built-in runserver mode (which was also due to lack of experience), and it was easy to encounter lagging situations. Later, we ran several processes using Gunicorn, and the language-level lagging issues disappeared. When it was slow, it was mostly due to high database load or MongoDB data aggregation.

The server configuration was not high, only 2C8G, running Python Web, MySQL, MongoDB, and several other application processes. This system ran for three years, but in the fourth year, due to changes in government relations, it was redeveloped by another company, which had fewer functions and was less user-friendly than ours, and I don’t know what language they used.

On this project, the combination of Python + MongoDB provided us with great flexibility, as the data filled out each year varied, and the statistical indicators were different. The entire system supported custom form filling, data validation, data import/export, custom statistics, etc. I feel it would be very difficult to achieve such results with another language, or achieving the same results would incur a higher cost.

Of course, this system had little maintenance involved; it was basically a one-time development, and afterwards, it just needed to ensure accessibility. At that time, I led a junior programmer in the development; I was responsible for the core architecture and most of the code implementation, while he handled simpler logic, UI, form definitions, etc. He might not have found it easy to understand the code I wrote. The maintainability of such complex system code largely depends on norms, documentation, and training, rather than language-level type constraints.

We also developed an internal office system for travel agencies, mainly targeting Southeast Asian travel agencies, supporting multiple languages and currencies, covering almost all daily operations of travel agencies, including planning, group formation, hotel transportation, shopping, guides, guests, accounting, revenue, finance, reports, charts, etc.

This was also done using Python 2 + Django 1.8. We deployed an independent web process and a database for each travel agency (the database name is independent, but one MySQL instance runs on one machine). Each web process consumes about 170MB of memory after startup. We used a 2C8G machine, and each machine could serve about 40 clients. Generally, the daily user data usage for each client is about 10, while larger travel agencies may have 20-30 employees operating simultaneously, with the concurrency for most clients estimated not to exceed 10.

At the beginning of each month, when each agency was doing accounting and exporting data, they occasionally reported lagging issues. From my observations, most of these were due to database performance issues. Our solution was to tell clients to wait a while before exporting (or if they had a lot of data to export, we would let them do it at night). As long as a few agencies staggered their data exports, there were no issues.

To save costs, we also self-built the database on cloud servers, basically pushing the cloud servers to their limits. During the day, the servers were generally running at over 80% CPU usage and over 90% memory usage, and the CPU would max out during data exports.

Before 2020, we had over 100 clients; during the three years of the pandemic, we basically did not engage in tourism. The last couple of years, we have regained some clients, but it is nowhere near what it used to be, as income has plummeted, and clients’ willingness to pay has decreased significantly.

I started using Python around 2012, after using Java and C# for a long time. Java was used for Android and web development, while C# was used for Windows desktop applications and Windows Phone development.

Before, I found the XML configuration in Java’s SSH framework quite cumbersome, and I don’t know if Spring is more convenient now. Besides the trouble with the framework, I feel that the Java language itself is relatively complex.

I find C# to be better than Java, but it falls short in terms of cross-platform capabilities. Therefore, now I only choose C# for desktop applications; otherwise, I won’t choose it. Besides, most of the time, desktop applications are also written using web technologies.

Python, on the other hand, feels simple enough to allow people to focus on business, which is why the company decided to use Python as the main language (even though at that time, I was not as familiar with Python as I was with Java and C#). The entire team was built around Python, but I also want to express that with the development of AI, Python will become even more popular.

In previous years, it was relatively difficult to recruit Python developers; most came from other languages and gradually adapted and became familiar. In recent years, there have been more people with Python backgrounds (thanks to public account advertisements?), but most of their levels are not high and still require familiarization and strengthening processes.

Generally, good developers can become quite proficient after half a year of working on a project, while slower ones may take over a year. The key factor is still interest; some people just have a natural inclination for programming and will work on personal projects after hours, which leads to rapid progress.

Different team experiences cannot be entirely replicated. I am one of the founders of the company, and I basically determine the technical direction, so there are no issues with turnover.

I am still very interested in using Python to solve most technical problems we encounter, including how to establish norms and train people to improve code control and personnel development.

Overall, over the years, I have accumulated some technical and management experience through ups and downs, and honestly, I feel less confident when it comes to switching to another language.