Author｜Jake Edge
Editor｜Natalie, Vincent, Debra
Source丨 AI Frontline WeChat Official Account
In recent years, the adoption of Python 3 has significantly increased, but there is still a long way to go. Large companies that adopt Python tend to run a lot of Python 2.7 code in their infrastructure, and Facebook is no exception. At this year’s PyCon 2018 conference, Facebook product engineer Jason Fried discussed the journey of how Python 3 went from being almost ignored to becoming the mainstream version of Python at the company over the past four years, showcasing Fried’s persistence as an engineer.
Here is the video of the talk:
Jason Fried, currently a product engineer at Facebook, played a key role in helping the company achieve this goal. He discussed some ideas on how to address the migration of Python versions in his talk.
Fried joined Facebook in 2011 and quickly realized he needed to self-learn Python, as Python code was easier to pass code reviews at Facebook. Later, he found himself becoming a major driving force behind Facebook’s adoption of Python 3. He stated that he never specifically planned for this; it was simply a natural outcome of using Python more.
(Jason Fried)
Jason Fried initially gained recognition for being very active in the internal Python community, often being the first to step up to answer questions. He gradually became known (or perhaps “notorious”) at Facebook as a supporter of Python, as he would directly modify Python code without permission when he saw issues. This worked well at Facebook, as there was no real top-down control mechanism; everyone had the right to make changes to a code change, just as you had the right to make code changes. Over time, he built credibility within Facebook’s internal Python community, which greatly aided his later efforts to lead the migration of Python versions at Facebook.
This is the complete timeline mentioned in Fried’s talk about how Python 3 went from being ignored to dominating at Facebook, showing that this process took nearly five years and was quite challenging.
2013 (Initial Support → Negative Sentiment → Glimmer of Hope)
Python 3 Will Never Appear on Facebook
The implementation of Python 3 at Facebook was very difficult, initially facing internal denial, leading Fried to believe it would never appear at Facebook, until now it has over 55% adoption rate, a very bumpy process.
He stated that changing Python versions at a “Facebook scale” takes a considerable amount of time and requires a lot of “diplomatic” efforts. He recounted how he and several engineers utilized their spare time to make Python 3 the main version at Facebook without any authority.
In 2013, Facebook planned to start preliminary support for Python 3 because they needed to add Python 3 support to the build system. However, since the Facebook library did not support Python 3, they could not add Python 3 to the build system. If the build system did not support Python 3, the Facebook library could not support Python 3. This was like the paradoxical military regulation described in “Catch-22”; Python 3 was “available” but received no support in the Facebook environment.
Additionally, in 2013, there was a significant amount of negative sentiment towards Python 3 within Facebook. Overall, they believed the company’s programming language would forever remain at Python 2.7. Some even suggested completely switching to another language. Fried had also stated (in the internal community) that Python 3 would never appear at Facebook. Only one person questioned him and suggested he do something to change this situation; although he ignored this suggestion at the time, the idea lingered in his mind.
Glimmer of Hope
In 2013, things began to change. In January of that year, the “linter” tool that Facebook was using required importing future to extend the lifespan of the Python 2 codebase by importing print_function, division, absolute_imports, and unicode_literals. They imported these packages wherever there were linter prompts, making it easier to convert modules to Python 3.
The Apache Thrift framework used for serialization and remote procedure calls was “ubiquitous” at Facebook. Since it only supported Python 2, it became the biggest obstacle. However, a survey initiated by the Facebook Thrift team regarding new Thrift features showed that developers generally wanted to add Python 3 support. Fried voted in favor, but not just to follow the trend; he believed the Python 2 interface needed to be restructured as it looked like Java.
When he saw Guido van Rossum discussing something called “Tulip” (which eventually became the asyncio module) at Yelp in San Francisco, his thoughts began to shift. He had always been a fan of asynchronous programming in Python, but it had become fragmented due to differences between frameworks (like Twisted and gevent). Tulip allowed interoperability between asynchronous I/O operations. Before the end of that talk, he communicated with the Facebook Thrift team, suggesting that Thrift should directly support Tulip instead of waiting for Twisted, gevent, and other frameworks to migrate to Python 3. A few days later, the Thrift team released a roadmap that included support for Python 3 and Tulip.
The Thrift team launched these two new features in early 2014, but there was no activity for the next six months. Users did not respond; in fact, they did not care and were even unaware that these changes had occurred. Fried humorously cited the example of a house built in China that no one lived in to illustrate this situation, which was quite ironic.
2014 (Cultural Change → Starting from Scratch → Enforced Implementation)
New Projects
In August 2014, he began rewriting a service and planned to use gevent and Python 2, but he later realized that if he did so, it would be outdated by the time he completed the project. To make a change, someone needed to be the first to make a change. That person had to be Fried.
So he started his project using Python 3, and as expected, he faced a “messy” situation. At that time, no one at Facebook was using Python 3, the build system did not support his code, and all third-party packages were only compatible with Python 2. After he fixed all the issues to get the code to compile, he encountered runtime issues.
To get the code to run properly, he had to fix all the problems. He rebuilt hundreds of third-party packages so that they could support both versions of Python simultaneously, and he had to ensure that all internal libraries were compatible with both Python 2 and Python 3. However, every day someone would submit changes in Python 2 to his dependencies. He had to keep fixing issues and grew weary of it. One solution was to enforce Python 3 compliance internally, but that was simply not possible at Facebook. However, if you acted as if you had some authority, people would gradually believe you really had that authority.
He leveraged many relationships to add Pyflakes (a lint tool) to the build process. He was able to justify adding it because, although PEP 8 existed, Pyflakes could address additional code quality issues. Moreover, Pyflakes had very few false positives, so it wouldn’t irritate developers. He set up Pyflakes to scan all code that needed review, first for Python 2 and then for Python 3. This helped extend Python 3 compatibility to all developers, not just himself, allowing his project to make progress.
Initially, he had to spend a lot of time explaining that “linter is not wrong” and that getting code to run on Python 3 was valuable. If developers began to feel that migrating to Python 3 was a difficult task, they would revert to the mindset of “let’s stay on Python 2 forever.” He had to ensure that developers could smoothly run code on Python 3.
2015 (Training)
Training
Although some difficulties were overcome, progress in expanding Python 3’s presence at Facebook was minimal or nonexistent. He joined the team that provided Python programming training for new employees at Facebook. He hoped that compatible code would only be used for legacy projects, while new projects should be developed using Python 3.
In 2015, he modified the Python training content for new employees, stating that Facebook would eventually transition to Python 3, and writing code only in Python 2 was pointless because it would need to be rewritten in the future. He taught new employees that all code should be consistent with Facebook’s infrastructure and build system; if not, they should report bugs or try to fix them themselves. This way, new employees began using Python 3 in their work, marking the beginning of progress. “Strangely, things just happened this way.”
In January 2015, he finally delivered his project. He spent more than half a year telling people how great it was and why they should use Python 3 as much as possible. Over the year, many allies at Facebook dedicated to promoting Python 3 became well-known within the company.
2016 (Python 3 Becomes the Default Programming Language)
One of those allies was Łukasz Langa, who “convinced Instagram to switch to Python 3.” In 2016, Fried and Langa formed a brand new team at Facebook to train Python internally, which they called “The Ministry of Silly Walks.” Although there were only two people, it was still a “Python team,” and thus the “authority” he mentioned earlier began to take effect: people believed they could make decisions about Python at Facebook.
In 2016, he noticed that the adoption of Python 3 was growing slowly but steadily. People mentioned it at meetings, and he often heard of new projects using it. Even though Python 3 was not the default setting, projects would choose to use it, and Facebook’s view of Python 3 had changed by this time. In May 2016, Fried expressed his intention to switch the build system to default to Python 3, and his proposal received almost unanimous support. A few days later, he completed the switch, and it did not bring any adverse effects.
Fried stated that in 2016, only ten people were pushing Python 3 projects at Facebook, with three being the main drivers, and there was constant personnel turnover; many people working on this project were part-time.
By the end of 2016, a project team published an article detailing the results of switching to Python 3. Developers found that when transitioning from Python 2 to Python 3, they only needed to make a few fixes, and the speed of running the code improved by 40%, using only half the memory. This debunked a rumor Fried had heard that Python 3 was slower than Python 2. Early versions of Python 3 might have been, but that was certainly not the case now, he said.
2017 (Instagram Migration)
Good Things Happened
In early 2017, Facebook basked in the glory of the Python 3 migration completed by Instagram. The upgrade of the Python version turned out not to be scary; instead, it brought usable new features. Facebook developers began using new static types or using asyncio to refactor old services. “Python became fun again at Facebook.”
The current question is when everyone can stop supporting Python 2. When regressions occur in libraries or modules that support Python 2, developers often ask if they can directly upgrade to Python 3. A few years ago, the situation was completely the opposite. “Oh, what a wonderful world!”
2018 (Python 3 Adoption Exceeds 55%)
He presented a chart showing the change in the percentage of Python service entry points at Facebook over time, starting from the third quarter of 2015, when there were only four Python 3 service entry points. By mid-2016, when the switch to default Python 3 occurred, Facebook had 4% of service entry points using Python 3. By March 2018, this percentage exceeded 50%. By mid-May, when he gave his talk, the proportion of Facebook service entry points running Python 3 had reached 55%. Code that could only run on Python 2 at Facebook was now in an awkward position, Fried stated.
Łukasz Langa tweeted, praising Python 3’s low CPU usage and speed improvements.
As the talk drew to a close, he summarized his presentation. Overall, his recommendations included:
-
What you need to do is innovate and make changes; results will naturally follow;
-
You must lead developers by “showing them the changes you want to see”;
-
You should also seek help from others; do not go it alone;
-
Additionally, training new employees to achieve your future goals is important.
-
Collect the necessary data;
-
Enjoy the results and write some “awesome things” in Python 3.
Finally, he answered some questions from the audience. Someone asked how to achieve the goals mentioned in the talk within traditional, hierarchical organizations. Fried believed that it might actually be easier because you do not need to convince thousands of developers; you just need to make management aware of the benefits. If it is difficult in a culturally conservative organization, focusing on code quality improvements might help. Another question was about the overall code rather than multiple entry points; for this, Fried suggested looking at the Instagram keynote at PyCon 2017 (see the beginning of the article).
The entire talk was very enlightening, including Fried’s emphasis on the importance of advocates and leaders, as well as the persistence required in a project.
Original link:
https://lwn.net/SubscriberLink/758159/f1f631e1535ab9d6/
Today’s Recommended Article
Click the image below to read
Why has Python developed so rapidly?
Today’s Benefit
Are you still struggling to learn Python on your own? Are you still troubled by not understanding the differences between Python 2 and Python 3? Are there some small tools that could be solved with just a few lines of code, but you waste a lot of time and effort because you don’t understand Python?
Learn Python from a Python expert with over ten years of experience, starting from zero, based on Python 3, and coding through practical problem scenarios to systematically understand and master Python!
Scan the code or click “Read the original text” to preview or subscribe
