First, let me introduce the software development process in my work.
Our company produces a physical product, and the software part of the product is developed and maintained by multiple departments.
The code from our team is compiled into static libraries (.a) or dynamic libraries (.so) depending on the product form, and together with libraries from other teams, they constitute a process that runs in linux user space. The entire product includes many processes, libraries, and configurations.
The code from our team calls functions provided by other modules within the same process and also provides function call interfaces to the outside.
When generating all software packages for the product, all modules upload their code to the specified branch in git, and through automated scripts, all processes, libraries, and configurations in the product are compiled. This process is also called pulling the pipeline.
If testing one’s own code is needed, unit tests (which do not depend on other modules) can be run locally, or tests can be conducted in a real environment. The real environment consists of a set of physical machines or a simulator (virtual machine) that simulates physical machines, where all software of the product is deployed, functioning identically to the product delivered to users.
Since unit tests can only verify part of the functionality, most of the time, validation still needs to be done in the real environment.
To validate the modified code, compile it into .a or .so, replace the previous library in the real environment, relink, and restart the process.
There are two ways to compile .a or .so: one is to compile locally using a pre-written docker compilation environment and compilation scripts. The local compilation script provided by the company for .a or .so always recompiles, taking about ten minutes.
The other way is the aforementioned pipeline, which involves uploading one’s code to git to generate a large software package for the product. This package can be directly deployed to the environment for validation, or the .a or .so within the package can be extracted and validated using the replacement method mentioned earlier. Compiling through the pipeline takes about half an hour.
Generally, local compilation is only supported when the module code can independently compile into .a or .so. Some product forms require the code from many modules to be compiled into one .so, in which case only the pipeline method can be used (each developer only has permission to their relevant module’s code and does not have permission to compile all modules needed to produce a .so).
Having become accustomed to modifying code and validating it with almost no waiting time, I now face a situation where I have to wait ten minutes or half an hour, forcing me to start thinking about compilation, library structure, assembly language, and other issues, hoping to improve the speed of turning code into .a or .so.
For local compilation, I have improved my speed significantly by studying the compilation scripts and understanding how to perform incremental compilation.
In cases where only pipeline compilation is possible, I wonder if I can directly modify the .so, that is, modify or add/remove assembly code within the .so. Alternatively, I could add a function with the same name in another .so that can be compiled locally to override the original function in the .so.
The code from our team is written in C, which makes it relatively convenient to perform some operations at the assembly or function symbol level.
Now, let me talk about my understanding of the compilation process from C language to assembly language.
When developing code in C, I often think that if I could intuitively feel the conversion process from C language to assembly language, I would have a clearer understanding of how the code I write operates, allowing me to view the code more calmly and peacefully.
According to textbooks, compilation consists of two steps: converting source code into intermediate code and converting intermediate code into machine code. Intermediate code can be understood as a ternary expression, similar to a=b+c.
I once wrote a program of over 200 lines in python that converts a custom-defined programming language similar to C into assembly language. Through this program, I realized that if performance is not considered, the compilation process of C language is quite understandable. That is, the conversion from source code to intermediate code is easy to grasp.
What I find difficult to understand is the conversion process from intermediate code to machine code. This is because there are many performance optimization steps involved, which require various techniques that I am not familiar with.
One way to become familiar with these techniques is to read textbooks. I did not take courses in this area during my studies, and now reading textbooks is also relatively slow for me.
Once, I thought that instead of starting with textbooks, I might as well look directly at the code of gcc or clang. Worst case, I could memorize the code, which would be equivalent to understanding the compilation process. However, when I actually tried to debug the gcc code, I found that fully understanding the code is even harder than reading textbooks!
I think if the optimization part of the compilation were done by AI, I wouldn’t be so concerned about the compilation process. I have done AI training and inference, and since the reasoning process of AI is beyond human understanding, people do not get caught up in the principles behind it. A huge table could be created to record which compilation optimization method AI uses in each case, and during compilation, it would execute according to this table. This table would be so large that no one could remember it all.
However, I know that compilers are not implemented using AI; they are based on principles that humans can understand as written in textbooks.
I also think that besides compiler optimizations, CPU instruction execution will also optimize. CPU optimizations might be implemented using AI, perhaps using the table lookup method mentioned earlier.
The final running effect of a program is the combination of compiler optimizations and CPU optimizations. If part of it is done by AI, it would not be much different from being entirely done by AI. Thinking this way, I am less troubled by the implementation of compiler optimization steps. Once I step out of this dilemma, I feel like I can better appreciate the conversion process from C to assembly language.
Of course, I will still take the time to study compiler principles, hoping that one day I can truly master the principles of compiler optimization.