Introduction AI is developing at an unprecedented speed, with new opportunities constantly emerging. If you wish to engage deeply with technical experts, product managers, and entrepreneurs to explore how AI is transforming various industries, feel free to scan the QR code at the end of this article to join the “AI Think Tank” group chat, where you can learn, think, and create together with like-minded partners!
I am ready to dive into the debate over programming languages. But first, let me state the most important point: use the tool you are most familiar with. For example, if you are comfortable with Python, then use Python. Additionally, use the tool that is most suitable for the task at hand. If that happens to be Python, then that’s perfectly fine.
Moreover, if you are already using a particular tool daily, it is completely normal to use it for other tasks as well. If you use a hammer to drive nails all day, it’s also acceptable to use it to open a beer or scratch your back. Similarly, if you write Python all day, it’s fine to use it to do some mixed linear modeling as well.
If you find it easy to use, then continue using it. But if you increasingly feel awkward and think that some tasks shouldn’t be this difficult, then this article might be for you.
First, I propose a viewpoint—people are overly reliant on the idea that “Python is the language of data science.” Its limitations are quite apparent. For many data science tasks, I would prefer to use R rather than Python.
The reason Python is so popular in the field of data science, in my opinion, is more due to historical coincidence and a kind of generality that is “good enough” across various aspects, rather than any inherent advantage it has in data science.
At the same time, I believe Python does perform well in deep learning. There is a reason PyTorch has become the industry standard. The data science I am referring to does not include deep learning, but rather other tasks: data cleaning, exploratory analysis, visualization, statistical modeling, etc.
As mentioned at the beginning, if you have sufficient reason to use Python every day (for example, training AI models), then it’s fine to use it to complete the remaining tasks as well. I do this myself when teaching deep learning courses. However, this does not negate the clumsiness and inconvenience I often feel when doing data analysis in the Python world.
Observations from the Frontline
Let me first share my personal experience without any analysis of the reasons. I have led a computational biology lab for over twenty years, during which I have collaborated with about thirty highly capable graduate students and postdocs. The rule in the lab is: everyone can freely choose any programming language and tools they want, and I never interfere.
The result is often that most people choose Python.
Here’s a situation I frequently encounter: a student comes to my office to show me some results. I say, “This is great, can you quickly plot the graph in a different way?” or “Can you quickly calculate a quantity I just thought of and plot it for me?” Such requests are usually tasks I know can be done in R in a few minutes, like changing a box plot to a violin plot, changing a line plot to a heatmap, changing a histogram to a density estimate, or calculating sorted values instead of raw values, etc.
Almost without exception, students using Python always respond, “This will take some time; I need to go back and figure it out, and I’ll let you know when it’s done.”
I want to emphasize that these students are all very capable. The issue is not that they don’t understand the tools, but rather that the tools themselves seem to be quite clumsy, making what I perceive as a “very simple request” often turn out to be anything but simple.
Whatever the reason, I have to conclude that Python has structural issues in certain fundamental aspects of data analysis. It could be a problem with the language itself, a problem with the library ecosystem, or a combination of both, but the result is real, and I have seen similar situations for a long time.
In fact, let me give another example to avoid you thinking, “This is a student-level issue.”
Last fall, I co-taught an AI course in biology with a seasoned data scientist. He does all his work in Python and is extremely familiar with NumPy, pandas, and matplotlib. In class, he was responsible for guiding students through Python exercises while I handled the theory. This allowed me to observe how a Python expert solves various data analysis problems in real-time.
My reaction was often, “Why does it have to be so complicated?” Many things I believe could be written in R in a few lines become long and convoluted in Python. If I don’t spend a lot of time retraining my brain and adapting to a completely different programming mindset, I simply cannot write that code. This sense of unfamiliarity is not “strange but elegant,” but rather “strange, awkward, and cumbersome.”
And I am quite sure the issue does not lie with my colleague; he is very capable. The real problem seems to be in the underlying design of these tools.
What is the Right Language for Data Science?
To take a step back, there are some basic considerations when choosing a language for data science. Here, data science refers to analyzing and summarizing data, finding patterns, fitting models, and creating visualizations—essentially all the work that researchers do in their daily data analysis. This is different from data engineering or application development.
Data science heavily relies on interactive exploration, requiring a lot of rapid experimental analysis. Therefore, a suitable language for data science must support interpreted execution and be usable in an interactive shell or notebook. Performance is secondary. When you are doing a linear regression, you don’t care if it takes 50ms or 500ms; what matters is whether you can immediately open the shell, write a few lines of code, and get results within minutes, rather than creating a new project, writing a bunch of template code, pleasing the compiler, and spending more time waiting for compilation than for execution.
Since interactivity and low startup costs are essential, the candidates naturally fall to scripting or data science languages like Python, R, Matlab, and Mathematica. Julia is also included, but to be honest, I don’t know enough about Julia to evaluate it. It may be excellent, but I have also seen many deep users express reservations about it. I won’t elaborate on that here. Commercial languages like Matlab and Mathematica, as well as Octave, which lacks an ecosystem, can also be set aside for now. The realistic choices are R and Python.
Before continuing, I want to add my thoughts on performance. Performance often comes at the cost of sacrificing other language features: additional burdens on programmers (like Rust), higher risks of hidden bugs (like C), or both. In data science, high bug risk is unacceptable, and programmer convenience is far more important than extreme performance. Computers are fast enough, and thinking can be painful. I would rather spend less effort telling the computer what to do and wait a bit longer. When there are real performance bottlenecks, I can always rewrite the critical path in Rust after clarifying the requirements.
Separating “Logic” from “Manual Labor”
The key to not making analysis more difficult is to separate “analytical logic” from “implementation details.” I want to write analysis code using conceptual descriptions: how the data should be processed, what the goals are; without being forced to worry about low-level details like data types, numeric indexing, loops, or manually assembling and disassembling data tables.
As soon as I start managing these details, I am likely being bogged down by “tedious mechanical labor.”
For example, let’s look at the Palmer Penguins dataset. It contains three species of penguins distributed across three islands. Suppose I want to calculate the mean and standard deviation of body weight across the two dimensions of “island × species,” excluding records with missing body weight.
The ideal data science language should allow me to complete this task in a way that is close to natural language, with the amount of code used being roughly equivalent to the vocabulary of the sentence I just described in English. Both R and Python can do this—only the difficulty level varies significantly.
Here’s the R code using the tidyverse style:
library(tidyverse)library(palmerpenguins)penguins |> filter(!is.na(body_mass_g)) |> group_by(species, island) |> summarize( body_weight_mean = mean(body_mass_g), body_weight_sd = sd(body_mass_g) )Now let’s look at the equivalent Python code, using pandas to accomplish the same task:
import pandas as pdfrom palmerpenguins import load_penguinspenguins = load_penguins()(penguins .dropna(subset=['body_mass_g']) .groupby(['species', 'island']) .agg( body_weight_mean=('body_mass_g', 'mean'), body_weight_sd=('body_mass_g', 'std') ) .reset_index())Overall, these two examples are very similar. For an analysis task of this scale, Python does quite well. Personally, I find the readability of R code slightly better (note how many quotes and parentheses are in Python), but the difference is not significant.
The processes are also consistent: take the penguins dataset, remove records with missing body weight, group by “species × island,” and then calculate the mean and standard deviation for each group.
Now, let’s compare the equivalent version using only basic Python syntax, without relying on data processing libraries:
from palmerpenguins import load_penguinsimport mathpenguins = load_penguins()# Convert DataFrame to list of dictionariespenguins_list = penguins.to_dict('records')# Filter out rows where body_mass_g is missingfiltered = [row for row in penguins_list if not math.isnan(row['body_mass_g'])]# Group by species and islandgroups = {}for row in filtered: key = (row['species'], row['island']) if key not in groups: groups[key] = [] groups[key].append(row['body_mass_g'])# Calculate mean and standard deviation for each groupresults = []for (species, island), values in groups.items(): n = len(values) # Calculate mean mean = sum(values) / n # Calculate standard deviation variance = sum((x - mean) ** 2 for x in values) / (n - 1) std_dev = math.sqrt(variance) results.append({ 'species': species, 'island': island, 'body_weight_mean': mean, 'body_weight_sd': std_dev })# Sort results to match order used by pandasresults.sort(key=lambda x: (x['species'], x['island']))# Print resultsfor result in results: print(f"{result['species']:10} {result['island']:10} " f"Mean: {result['body_weight_mean']:7.2f} g, " f"SD: {result['body_weight_sd']:6.2f} g")This code is noticeably longer, filled with loops, and requires breaking down the dataset and reassembling it. Regardless of which language you prefer, it should be clear that a version that allows you to express what you want to do directly, without being bogged down by a bunch of trivial steps, is far superior.
In simple terms, I believe the reason Python code often turns into “manual labor” is that no matter how much you want to maintain a high-level abstraction and write cleanly, the language itself or its libraries will always complicate things, forcing you to deal with those low-level details.
(👇 Long press to recognize the QR code)
Source: CSDN