Excerpt from Stream
Author: Thierry Schellenbach
Translated by: Machine Heart
Contributors: Huang Xiaotian, Li Yazhou
Switching to a new programming language is often a significant decision, especially when only one member of your team has experience with it. This year, the Stream team transitioned its primary programming language from Python to Go. This article explains the nine main reasons behind this change and how to effectively manage the transition.
Why Use Go
Reason 1: Performance
Go is extremely fast. Its performance is comparable to Java or C++. In our usage, Go is generally 30 times faster than Python. Here is a benchmark comparison between Go and Java:
Reason 2: Language Performance Matters
For many applications, the programming language simply acts as glue between the application and the dataset. The performance of the language itself often seems irrelevant.
However, Stream is an API provider serving Fortune 500 companies and over 200 million end users. Over the years, we have optimized Cassandra, PostgreSQL, Redis, etc., but ultimately hit the limits of the language we were using.
Python is great, but it performs poorly in serialization/deserialization, sorting, and aggregation. We often encounter situations where Cassandra retrieves data in 1ms, but Python takes 10ms to convert it into an object.
Reason 3: Developer Efficiency & Avoiding Over-Innovation
Take a look at a snippet from the excellent introductory tutorial “Getting Started with Go” (http://howistart.org/posts/go/1/):
package main
type openWeatherMap struct{}
func (w openWeatherMap) temperature(city string) (float64, error) {
resp, err := http.Get("http://api.openweathermap.org/data/2.5/weather?APPID=YOUR_API_KEY&q=" + city)
if err != nil {
return 0, err
}
defer resp.Body.Close()
var d struct {
Main struct {
Kelvin float64 `json:"temp"`
} `json:"main"`
}
if err := json.NewDecoder(resp.Body).Decode(&d); err != nil {
return 0, err
}
log.Printf("openWeatherMap: %s: %.2f", city, d.Main.Kelvin)
return d.Main.Kelvin, nil
}
If you are a beginner, you would not be surprised by this code. It demonstrates various assignments, data structures, pointers, formatting, and the built-in HTTP library.
When I first started programming, I loved using Python’s higher-order features. Python allows you to creatively use the code you are writing, for example, you can:
-
Register classes using MetaClasses during code initialization
-
Override truthiness
-
Add functions to the built-in function list
-
Overload operators in wonderful ways
Undoubtedly, this code is fun, but it also makes it difficult to understand when reading others’ work.
Go forces you to stick to the basics, which makes it easier to read any code and quickly understand what is happening.
Note: Of course, how easy it is still depends on your use case. If you are creating a basic CRUD API, I would still recommend using Django + DRF or Rails.
Reason 4: Concurrency & Channels
Go as a language is dedicated to simplifying things. It does not introduce many new concepts but focuses on creating a simple language that is exceptionally fast and easy to use. Its only innovation is goroutines and channels. Goroutines are Go’s lightweight method for threading, while channels are the preferred way for communication between goroutines.
The cost of creating goroutines is very low, requiring only a few kilobytes of additional memory, which makes it possible to run hundreds or even thousands of goroutines simultaneously. You can use channels to implement communication between goroutines. The Go runtime can handle all the complexity. The goroutines and channel-based concurrency model makes it very easy to utilize all available CPU cores and handle concurrent I/O—all without complex development. Compared to Python/Java, running a function on a goroutine requires minimal boilerplate code. You just need to use the keyword “go” to add a function call:
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello")
}
Go’s concurrency model is very easy to grasp and is also more interesting compared to Node; in Node, developers must closely monitor the handling of asynchronous code.
Another excellent feature of concurrency is the race detector, which makes it easy to determine if there are race conditions in asynchronous code. Here are some great resources to get started with Go and channels:
-
https://gobyexample.com/channels
-
https://tour.golang.org/concurrency/2
-
http://guzalexander.com/2013/12/06/golang-channels-tutorial.html
-
https://www.golang-book.com/books/intro/10
-
https://www.goinggo.net/2014/02/the-nature-of-channels-in-go.html
Reason 5: Fast Compilation Time
Currently, the largest microservice we have written in Go compiles in just 6 seconds. Compared to the sluggish compilation speeds of Java and C++, Go’s fast compilation time is a major efficiency advantage. I love fencing, but it’s even better when things are done while I still remember what the code is supposed to do.
Compilation of code before Go
Reason 6: Building Team Capability
First, the most obvious point is that there are far fewer Go developers compared to older languages like C++ and Java. It is known that 38% of developers are familiar with Java, 19.3% with C++, and only 4.6% with Go. GitHub data shows a similar trend: Go is more popular than Erlang, Scala, and Elixir, but not as popular as Java and C++.
Fortunately, Go is very simple and easy to learn. It only provides basic functionality without excess. The new concepts introduced by Go are the “defer” statement and built-in concurrency management with goroutines and channels. Because of Go’s simplicity, any Python, Elixir, C++, Scala, or Java developer can form an efficient Go team within a month.
Reason 7: Strong Ecosystem
For a team of our size (about 20 people), the ecosystem is crucial. If you need to redo every feature, you cannot create value for customers. Go has strong tool support with stable libraries for Redis, RabbitMQ, PostgreSQL, template parsing, task scheduling, expression parsing, and RocksDB.
Go’s ecosystem has a significant advantage over languages like Rust and Elixir. Of course, it is slightly inferior to languages like Java, Python, or Node, but it is stable, and you will find high-quality packages available for many basic needs.
Reason 8: GOFMT, Enforced Code Formatting
Gofmt is a powerful command-line feature built into Go’s compiler to enforce code formatting. Functionally, it is similar to Python’s autopep8. Consistent formatting is important, but the actual formatting standards are not always very important. Gofmt formats code in an official style, avoiding unnecessary discussions.
Reason 9: gRPC and Protocol Buffers
Go has first-class support for protocol buffers and gRPC. These two tools work well together to build microservices that need to communicate via RPC. We only need to write a manifest to define what RPC calls occur and their parameters, and then server and client code can be automatically generated from that manifest. The generated code is not only fast but also has a very small network footprint.
From the same manifest, we can generate client code in different languages, such as C++, Java, Python, and Ruby. Therefore, the internal communication RESET endpoints do not diverge, and we only need to write almost identical client and server code each time.
Disadvantages of Using Go
Disadvantage 1: Lack of Frameworks
Go does not have a major framework like Ruby’s Rails, Python’s Django, or PHP’s Laravel. This is a hotly debated issue in the Go community, as many believe we should not start with frameworks. In many cases, this is indeed true, but if you just want to build a simple CRUD API, using Django/DRF, Rails, or Phoenix would be much simpler.
Disadvantage 2: Error Handling
Go helps developers handle compilation errors simply by returning errors (or call stacks) through functions and expected call codes. While this method is effective, it is easy to lose the scope of where the error occurred, making it difficult to provide meaningful error messages to users. The errors package allows us to add context and stack traces to returned errors to address this issue.
Another problem is that we may forget to handle errors. Static analysis tools like errcheck and megacheck can help avoid these oversights. While these solutions are effective, they may not be the most correct approach.
Disadvantage 3: Package Management
Go’s package management is far from perfect. By default, it does not allow for specifying specific versions of dependencies or creating reproducible builds. In contrast, Python, Node, and Ruby have better package management systems. However, with the right tools, Go’s package management can perform well.
We can use Dep to manage dependencies, which can also specify specific package versions. Additionally, we can use an open-source tool called VirtualGo, which can easily manage multiple projects written in Go.
Python vs Go
One interesting experiment we conducted was to write a ranking feed in Python and then rewrite it in Go. Here is an example of the sorting method:
{
"functions": {
"simple_gauss": {
"base": "decay_gauss",
"scale": "5d",
"offset": "1d",
"decay": "0.3"
},
"popularity_gauss": {
"base": "decay_gauss",
"scale": "100",
"offset": "5",
"decay": "0.5"
}
},
"defaults": {
"popularity": 1
},
"score": "simple_gauss(time)*popularity"
}
Both Python and Go code need to meet the following requirements to support the above sorting method:
-
Parse the score expression. In this example, we want to transform the string simple_gauss(time)*popularity into a function that can take activity as input and return a score as output.
-
Create partial functions on the JSON config. For example, we want “simple_gauss” to call “decay_gauss” with the key-value pairs “scale”: “5d”, “offset”: “1d”, “decay”: “0.3”.
-
Parse the “defaults” configuration to provide feedback in cases where a specific domain is not clearly defined.
-
Start using the function from step 1 to score all activities in the feed.
Developing the Python version of the sorting code took about 3 days, including writing code, testing, and documentation. After that, we spent about 2 weeks optimizing the code. One optimization was to translate the score expression simple_gauss(time)*popularity into an abstract syntax tree. We also implemented caching logic, which would pre-calculate the score each time.
In contrast, developing the Go version of the code took 4 days, but no further optimization was needed. So while the initial development in Python was faster, Go ultimately required less work. Additionally, Go code was over 40 times faster than highly optimized Python code.
This is just one of the benefits we experienced from switching to Go. Of course, comparisons cannot be made like this:
-
This sorting code was my first project written in Go;
-
The Go code was written after the Python code, so I had prior understanding of the case;
-
The quality of Go’s expression parsing library is superior.
Elixir vs Go
Another language we evaluated was Elixir. Elixir is built on the Erlang virtual machine. It is a fascinating language, and we considered it because one of our team members has extensive experience with Erlang.
In our use case, we observed that Go’s raw performance was better. Both Go and Elixir handle thousands of concurrent requests well; however, for individual requests, Go is actually faster. Another reason we chose Go over Elixir was the ecosystem. The libraries for the components we needed in Go are more mature. In many cases, Elixir libraries are not suitable for production use. Additionally, it is challenging to find/train developers who use Elixir.
Conclusion
Go is a highly efficient language that strongly supports concurrency. At the same time, it is as fast as C++ and Java. Although building things with Go takes more time compared to Python and Ruby, it can save a lot of time in subsequent code optimization. At Stream, we have a small development team providing feed streams for 200 million end users. For novice developers, Go combines a strong ecosystem, ease of use, super-fast performance, and highly supported concurrency, making it a good choice. Stream still uses Python for personalized feeds, but all performance-intensive code will be written in Go.
Original article link: https://getstream.io/blog/switched-python-go/
The Baidu AI Practical Camp in Shenzhen will be held on October 19 at the Shenzhen Kexing Science Park International Conference Center. AI developers and technology practitioners hoping to enter the AI field, please click “Read the original” to register and join Baidu in creating the era of artificial intelligence.