Why Are There No Embedded Software Architects?

This embedded system specifically refers to those based on the Linux platform; microcontrollers and other RTOS are not discussed.

I have been engaged in embedded software development for 6 or 7 years, involving BSP, drivers, application software, Android HAL, frameworks, etc. Besides focusing on the development of the embedded industry, I also pay some attention to technologies in the web, backend services, distributed systems, etc.

Recently, I have been thinking about changing my career direction to backend server development. However, since there was no practical demand for this in my previous job, I only followed some knowledge on my own, such as NIO, epoll, Nginx, ZeroMQ, libevent, libuv, high concurrency, distributed systems, Redis, Python, Tornado, Django. My understanding is quite superficial and not in-depth. Unexpectedly, I have repeatedly been looked down upon by the internet industry, and opportunities for interviews are few and far between.

This makes me wonder what the problem is. Has the embedded background become so undesirable? Back then, embedded systems and driver development were highly sought after (a bit of an exaggeration, but 8 or 9 years ago, embedded systems were indeed considered more prestigious than Java web development).

Problems always have their reasons, and I would like to share my understanding:

Is embedded really prestigious? Why are there no embedded software architects?

When I open various job websites and search for architects, I find various system architects, web architects, backend service architects, etc., but it is very difficult to find embedded software architects. Does embedded software not need architecture? Do drivers not need architecture? The answer is of course they do, but why are there no positions for this?

In my opinion, currently, embedded development in China is mainly divided into embedded low-level development and embedded application development. Low-level embedded development is generally referred to as driver development or BSP development, and sometimes it’s called Linux kernel development, which sounds quite prestigious.

But why is there no architect for such a prestigious name? The architects of Linux and kernel are Linus and other Linux kernel developers. The Linux kernel or operating system itself is a general platform that solves common problems, and the big names in the open-source community have already established architectural rules, leaving little room for creativity. Most of the work only requires filling in according to the established rules.

For most companies in China, the business needs are just integrating peripheral devices, porting embedded platforms, building and trimming, and the business needs will not exceed the functionality provided by the kernel, resulting in no new architecture that developers need to design and implement. So what are embedded BSP developers doing? Besides debugging various peripherals and cleaning up after hardware, they are just solving stability bugs (I won’t go into detail about specific work here; debugging peripherals only increases experience and breadth, contributing little to depth. It just follows the path of not being able to debug -> able to debug -> debugging faster, while solving stability issues requires some accumulated experience).

As for embedded application development, the business logic is generally quite simple and is often overlooked, so employers feel there is no need to look for architect-level personnel.

Thus, it seems that the embedded industry indeed does not require architects, and being looked down upon by the internet industry is not surprising.

But is it really the case? For low-level embedded development, there are probably not many developers in China who can propose architectural optimizations for the kernel and driver architecture, so for most ordinary people, it is better not to “fantasize” about becoming a Linux kernel architect (though I believe there are certainly capable experts among the Chinese). It is more reliable to discover and solve some bugs.

So for embedded application layer development, do we really not need architecture?

Based on my actual experience, I would like to discuss the architectural design and optimization of an embedded device application software I once took over: I once took over a project that used a single-process multi-thread model, including several modules represented by a, b, c, d, e. The business logic of this project determines that these modules are quite interrelated.

For example, in the initial design, the module a was a state monitoring module, which would call the interfaces of modules b and c based on the monitored state to implement some functions (the advantage of multi-threading is that direct calls are very convenient, so most developers do this, which is simple and crude). However, requirements are always changing, and a new module f is added, which also needs to process the state monitored by module a. According to the previous approach, to complete this function, two steps are needed:

1. Provide an interface in module f.

2. Call that interface in module a, and thus the new requirement has been “perfectly” solved.

As mentioned earlier, requirements are always changing, and a new customized requirement arises, requiring the addition of another module g, which also processes the state monitored by module a. However, this customized requirement does not need the recently added module f. The simplest and crudest way to handle this is to define a macro to distinguish between the customized requirement and the previous general requirement, building two versions of the program. This approach seems simple, but if the customized requirements gradually increase, maintaining so many customized versions of the program will become a nightmare, and code management and generality will also be significant issues, while the code will be filled with differentiated processing based on different macro definitions.

A better approach is to add dynamic monitoring of device model versions, using a single build program version to dynamically support all customized requirements, thus reducing the maintenance of different build programs. However, this approach only solves the maintenance of build program versions and does not address the issue of differentiated processing of macro definitions. If these differentiated judgments are concentrated in one place, it will not lead to significant complexity issues. However, it is obviously difficult to guarantee this; these differentiated processing may spread throughout the project, making maintenance a nightmare.

There is no need for profound software ideas; most people would think to extract the differentiated parts and manage them in a centralized location, focusing any modifications to this unified management location.

The general practice is to use callbacks to set hooks, and then customize differentiated requirements in the callback. In the above example, this means adding a hook in module a, and during system initialization, based on the different device version numbers, customizing the callback handling functions, while ensuring that these customized callback handling functions are processed in the same place; otherwise, it will still be meaningless if they are scattered across various corners (if hooks are not placed, these differentiated configurations cannot be gathered together). The benefit of this approach is that any changes to functional requirements will not affect the processing of module a, meaning we can add functionality without modifying the code of module a (the previous approach required modifying the calling process of module a), thus achieving a separation of modules.

At this point, the architecture of the second solution (which actually cannot be called architecture) has improved significantly compared to the first solution, at least making it easier for developers. For other customized requirements, developers only need to modify the callback handling, focusing on the differentiated parts.

Software needs to evolve continuously; is the second solution the optimal solution? Of course not, is there still room for optimization?

Before that, let’s digress and discuss the advantages and disadvantages of multi-thread/multi-process models, mainly focusing on the advantages of multi-process:

Without going into textbook explanations, I advocate the multi-process model for large projects, regardless of performance, mainly for the following reasons:

1. Decoupling of modules: Many developers maintaining multi-thread model projects should have encountered the following issues: direct calls between modules. If you don’t believe it, your project must be modular, right? Now randomly delete a module and try to build it; I believe in most cases, you will encounter situations where a certain function call or a global variable cannot be found, indicating that there is strong coupling between your modules.

Due to the inherent advantage of multi-threading, the mutual visibility of address spaces makes direct calls very easy, and many inexperienced engineers can easily write simple and crude interfaces for direct calls. If they encounter a static interface function, they will also conveniently remove the static keyword and use it directly. As the entire project continues to add functionality, the inter-module interactions increase, and the coupling becomes higher.

My advocacy for the multi-process approach is that it physically isolates this “convenient” communication method, forcing you to think more about whether the interaction between modules is necessary. If it is necessary, you will further consider whether the interface definition is clear and straightforward (because inter-process communication is relatively troublesome, developers will try to minimize interactions and clearly define interfaces; otherwise, they will be making things difficult for themselves). This is akin to life; if everything goes smoothly, people may not think too much, but if there are some bumps along the way, it leads to different insights.

Therefore, my belief is that the multi-process model will compel you to think more about program design and physically reduce module coupling.

Abstracting common components, separating common functionality from business logic: When modifying a multi-thread model to a multi-process model, it is often found that some interface codes are duplicated across multiple process modules because the interface functions were in a single process space, and everyone could call them directly. For example, if interface A is called by modules a and b, once modules a and b are separated into two independent processes, interface A needs to be implemented separately in both a and b. No need to explain; duplicate code is a big taboo in software engineering and must be eliminated. The solution is simple: separate these interfaces called by multiple modules into a library for other modules to call. After completing this work, you will notice that the interfaces you have stripped away can exist as common components for the entire project. Ideally, the code in the library is a general foundational component, while the modules are independent business processing modules.

2. Easier problem localization: In a multi-thread model, if a thread exits abnormally, it causes the entire process to exit. Of course, with some crash information, it can be determined which thread died. But if these thread modules are maintained by multiple teams or individuals, when the entire process crashes, how to determine which team should solve it becomes a significant issue. Moreover, sometimes the phenomenon occurs where one thread hangs, but it is actually caused by another thread module (the curse of coupling), leading to blame-shifting between teams (confident engineers believe their code has no issues).

However, if a multi-process model is adopted, if your service process crashes, you will have to find the cause yourself; there is nothing to argue about.

3. Easier performance testing: In a multi-thread model, it is not easy to check the resource usage of a single thread (at least some embedded systems do not have comprehensive commands). When the resource consumption of the entire process is high, it is difficult to pinpoint which module’s thread is the issue, as mentioned before. In contrast, if it is a multi-process model, whoever’s process consumes a lot of resources should investigate; this is essentially a granularity issue. The same system, divided into multiple processes, will have a single process complexity that is much lower than that of a single-process system, and with reduced complexity, it becomes easier to locate and diagnose various issues.

4. Distributed deployment: The internet industry has always emphasized distributed systems, cloud computing, etc., while the embedded industry has it tough; it seems there is no need for distribution. This is indeed true; in most cases, embedded systems use single chips running independently, and distributed scenarios are rare. However, what if one day you need to distribute the functionality originally handled by one chip across two chips? The multi-process expansion becomes much easier.

This is just a special example; in fact, embedded devices are inherently a distributed industry; they have already been separated from the beginning, rather than evolving from centralized to distributed systems.

Facilitating code permission isolation within companies: I actually despise this practice; companies should trust their employees, but given that trust in China has… some isolation is understandable.

In a multi-thread model, as mentioned earlier, if you remove a module, you may not even be able to build, so should all code be exposed to all engineers? Obviously not, so each module can only be provided in library form. However, I believe organizing common functional interfaces into a common library is a normal approach, while providing business-related modules as libraries is somewhat…

Thus, to summarize, all the aforementioned advantages are not particularly critical points; they do not provide multi-process models with absolute advantages over multi-thread models. It is just that from my perspective, the multi-process model forces engineers to think about solving certain issues (and experienced engineers will consider these issues regardless of the model).

Having said so much, we should consider how to modify the previous project example into a multi-process model; otherwise, it’s just theoretical.

The first issue to tackle is selecting the communication method for multi-processes; direct calls between threads can no longer be used. So how to choose the communication method for multi-processes?

Linux provides many IPC methods, which I won’t list here. For control with non-large data volumes, a good method is to use sockets. On the local machine, Unix sockets are often used (what’s the advantage of this? When you need to turn a single system into a distributed system, the advantages become clear).

However, merely using sockets to implement the functionality of the previous example will also encounter some issues. First, let’s clarify that the optimized second solution cannot continue to be used in the multi-process model for a simple reason that should not need explanation.

The simple approach is to base it on the first solution, changing direct calls to socket communication (just define the communication protocol). However, engineers familiar with socket development know that initiating socket communication requires some preliminary work (mainly establishing connections to associate the two modules). As a result, the previous example will look like this: module a needs to establish connections with modules b and c; if module f is added, module a must also connect with module f. When you sketch a connection diagram in your mind, it feels like we have woven a spider web, with complex interrelationships between nodes, and like the first solution, adding a module related to a requires modifying the code of module a, making this situation even more cumbersome and complicated than the multi-thread model. This approach is definitely a nightmare.

How to resolve this? Many people will likely think of adopting a bus distribution method. Those familiar with Android system development will think of Binder; those familiar with OpenWRT will think of Ubus; desktop developers will think of Dbus; and internet industry developers will certainly know the pub/sub module provided by Redis.

The principles of Binder, Ubus, etc., are quite simple: establish a message center and build a forwarding routing model, where all other modules do not interact directly but use the message center for forwarding and routing. How to determine the routing rules is done using the observer pattern for subscription/publishing (embedded developers or C language developers often mistakenly think that design patterns are associated with object-oriented languages and are unique to them; although many experts have popularized this, some developers’ information channels are quite closed, leading to this idea being quite prevalent).

Based on this model, the requirements of our previous example can be easily resolved. By adding a message center module, all modules needing communication only connect to this message center module and subscribe to events they are interested in. When an event occurs, they only need to handle it accordingly.

In this way, modules b and c can subscribe to events from module a. When module a detects a certain event, it publishes that event, which first reaches the message center and is then forwarded to modules b and c. For the newly added module f, it only needs to subscribe to events from module a, avoiding any need to modify the code of module a, thus making functionality extensions very convenient.

Moreover, the previously mentioned customization development has also been simplified. If a customized version needs to add module g, it only requires starting module g as an independent process in the customized version and subscribing to events from module a. The difference between the customized version and the general version lies in whether the process of module g is started, thus achieving a goal in software engineering: adding functionality is like building blocks; you only need to insert (start) or remove (not start) a module, and changes in functionality are confined to one or a few modules, having no impact on the main framework.

The above roughly describes the iterative optimization process for project requirements. The example seems based on embedded projects, but it appears applicable to software engineering as a whole.

Coming to the internet industry:

Examining the articles shared by architects of major websites about the technological transformations of their sites, the first mention is usually the decomposition of previous application server functionalities based on business, making them more granular (for example, splitting services for login, registration, transactions, products, sellers, etc. in e-commerce), and then deploying the split services across multiple servers to provide concurrency. Does this sound familiar? Is there a similarity to the division from multi-threading to multi-processing?

After decomposition, communication issues arise, and many message middleware solutions emerge, such as Alibaba’s Dubbo. A simple understanding of the principles of these middleware shows that they generally revolve around subscription and publishing, RPC mechanisms, and can be considered largely similar, while the challenges lie in protocol formulation and performance enhancement.

When comparing the load balancing solutions in the internet industry, it seems that the load balancer at the front end resembles a message center.

All of the above is meant to illustrate a point: software design is interconnected, and the underlying ideas are the same. Although the business logic in the embedded industry is relatively simple, upon careful consideration, there are still many architectural improvements and designs available.

However, it saddens me that some embedded developers, due to the simplicity of business logic, feel that using less than optimal approaches can still resolve issues and do not consider how to optimize and improve. For instance, in the example of the first solution, if there are not many customized requirements, maintaining it does not pose much of a problem. Even if many customized requirements arise, hiring some junior programmers can also manage it. Companies where one person is responsible for one set of code for a project do exist.

Likewise, there should not be an insurmountable wall between the internet and embedded industries; we should focus more on the universal principles of software engineering.

1.The country officially announces four major platforms for artificial intelligence; the future is here!

2.Linux engineers share insights: Six steps to getting started with embedded Linux.

3. A project experience of an electronic engineer from school to work; perhaps you can learn from it!

4. Knowledge post | Detailed explanation of the embedded Linux startup process.

5.To look like an expert in embedded development, you should know these!

6.Analysis of common issues with C51 microcontrollers!

Disclaimer: This article is a network reprint, and the copyright belongs to the original author. If there are any copyright issues, please contact us, and we will confirm the copyright based on the copyright certificate you provide and pay the remuneration or delete the content.