Why Are There No Software Architects in Embedded Systems?
Follow+Star Public Account, don’t miss exciting contentMaterial Source | NetworkWhen you open various recruitment websites and search for architects, you will find various system architects, web architects, backend service architects, etc., but it is very difficult to see embedded software architects. Does embedded software not need architecture? Does the driver not need architecture? The answer is of course it does, but why are there no positions in this area?Currently, embedded development in China is mainly divided into embedded low-level development and embedded application development. Low-level embedded development is generally called driver development or BSP development, and sometimes it is also referred to as Linux kernel development, which sounds quite impressive.Why is there no architect for such impressive names: The architect of the Linux kernel is Linus and a group of Linux kernel developers and maintainers. Because the Linux kernel or operating system itself is a general platform that solves common problems, the big names in the open-source Linux community have already established architectural rules, leaving little room for creativity. Most of the work only requires filling in according to the framework of rules.Moreover, with the current business needs of most companies in China, they are only integrating peripheral devices, porting embedded platforms, and building and trimming. The business requirements will not exceed the functional range provided by the kernel. This leads to no new architecture for developers to design and implement. So what are embedded BSP developers doing: Besides debugging various peripherals and cleaning up after hardware, they are just solving stability bugs (I won’t go into detail about specific work here; debugging peripherals only increases experience and breadth, contributing little to depth. It just follows the path of not knowing how to debug -> knowing how to debug -> debugging quickly, while solving stability issues indeed requires some accumulated experience)As for embedded application development, the business logic is generally simple and often overlooked, so recruiters feel there is no need to find architect-level candidates.Thus, it seems that the embedded industry indeed does not need architects, and being looked down upon by the internet industry is not surprising.But is it really like this? For low-level embedded development, there should not be many developers in China capable of proposing architectural optimizations for kernel and driver architecture, so for most ordinary people, it is better not to “fantasize” about becoming a Linux kernel architect(Of course, I believe there are definitely capable experts among our people) and to find and solve some bugs is more reliable.So do we really not need architecture for embedded application layer development?Here I will share my (the author’s) practical experience in the architectural design and optimization of an embedded device application software:I once took over a project that adopted a single-process multi-thread model, including several modules represented as a, b, c, d, e. The business logic of the project determined that these modules have many interrelations.For example: In the initial design, module a was a state monitoring module that would call the interfaces of modules b and c based on the monitored state to implement some functions (the advantage of multi-threading is that direct calls are very convenient, so developers mostly do this in a simple and crude way), but requirements are always changing, and a new module f needs to be added, which also needs to process the state monitored by module a. According to the previous approach, completing this function involves two steps: 1. Providing an interface in module f, 2. Calling that interface in module a. Thus, the new requirement has been “perfectly” solved.As mentioned earlier, requirements are always changing. A new requirement has come in: the customer proposes a custom requirement, needing to add another module g, which also processes the state monitored by module a, but this custom requirement does not require the recently added module f. At this point, the simplest and most crude way is to define a macro to distinguish this custom requirement from the previous general requirement, building two program versions. This approach seems simple, but if the custom requirements gradually increase, maintaining so many customized program versions becomes a nightmare, and code management and generality will also be significant issues. At the same time, the code will be filled with differentiated processing for different macro definitions. #ifdef xxx; do_something; #endif A better approach is to add dynamic monitoring of device model versions, using one build program version to dynamically support all custom requirements, thus reducing the maintenance of different build programs. However, this approach only solves the problem of maintaining build program versions and does not solve the issue of differentiated macro processing. It just changes the previous macro judgment to dynamic device version number judgment. If these differentiated judgments are concentrated in one place, it will not cause significant complexity issues, but it is evident that this is hard to guarantee, and these differentiated processes may spread to all corners of the project, making project maintenance a nightmare.You don’t need any profound software philosophy; most people will think of extracting the differentiated parts and placing them in a unified place for centralized management, and only modifying the differentiation in this unified management place.The common practice is to use callback settings to hook, and then customize the differentiated requirements in the callback. For the above example, this means adding a hook in module a, and during system initialization, customizing the callback processing function based on the different device version numbers, while ensuring that these customized callback processing functions are handled in the same place; otherwise, it is meaningless to still be scattered across various corners (the previous method without hooks cannot group these differentiated configurations together). This processing brings another benefit: changes to functional requirements will not affect the processing of module a, meaning we can add functionality without needing to modify the code of module a (the previous method required modifying the calling process of module a), thus achieving a separation of modules.Thus, the architecture of the second plan (which may not even be called architecture) has improved significantly compared to the first plan, at least making it slightly easier for developers. For other custom requirements, developers only need to modify this callback handling and focus on the differentiated parts.Software needs to evolve continuously. Is the second plan the optimal solution? Of course not; is there still room for optimization?Next, let’s digress and talk about the pros and cons of multi-threading/multi-processing models, mainly focusing on the advantages of multi-processing:I won’t mention the textbook explanations. Firstly, I advocate the multi-process model for large projects, regardless of performance, primarily for the following reasons:Decoupling of modules:Many developers maintaining multi-thread model projects must have encountered the following problems: direct calls between modules. If you don’t believe it, your project must be modular, right? Now randomly delete a module and see if it can build successfully (just build, no need to run). I believe in most cases, you will encounter a situation where a certain function call or global variable cannot be found, indicating strong coupling between your modules.Due to the inherent advantages of multi-threading, where the address spaces are mutually visible, direct calls become very easy. Many inexperienced engineers can easily write simple and crude interfaces for direct calls. If they encounter a static interface function, they might also remove the static for convenience and use it directly. As the entire project continues to add functionality, the intersections between modules increase, and the coupling becomes higher.The reason I advocate the multi-process model is that it physically isolates this “convenient” communication method, leading to more thought about whether such interactions are necessary. If necessary, they will further consider whether the interface definitions are simple and clear (because communication between processes is relatively cumbersome, developers will aim to reduce interactions and clarify interface definitions, or they will end up torturing themselves). This is like life: if everything goes smoothly, people may not think too much, but if there are some bumps along the way, there will be a different kind of insight.Thus, my thought is that the multi-process model forces you to think more about program design and physically reduces the coupling of modules.Abstracting common components, separating common functionality from business logic: When modifying a multi-thread model to a multi-process model, it is often found that some interface code appears repeatedly in multiple process modules because the interface functions were in one process space, and everyone could call them directly. For example, if interface A is called by modules a and b, once modules a and b are separated into two independent processes, interface A needs to be implemented in both a and b. No need to explain; duplicate code is a taboo in software engineering and must be eliminated. The approach is simple: separate these interfaces called by multiple modules into a library for other modules to call. When you complete this part of the work, what do you find? The separated interfaces can serve as common components for the entire project. Ideally, the code in the library is a common foundational component, while the modules contain independent business processing modules.Facilitating problem location:In a multi-thread model, when a thread exits abnormally, it causes the entire process to exit. Although crash information can help identify which thread died, if these thread modules are maintained by multiple teams, when the entire process crashes, how to determine which group should resolve it will be a significant problem. Sometimes, a thread may hang, but it is actually caused by another thread module (the curse of coupling). When encountering such situations, it is inevitable that disputes and finger-pointing will arise between teams (confident engineers believe their code is perfect).However, if the multi-process model is adopted, well, your service process crashed; you find the reason yourself, and there’s nothing to argue about.Facilitating performance testing:In a multi-thread model, the resource usage of a single thread is not easy to check (at least there are some embedded systems without complete commands). When the entire process consumes a lot of resources, how to determine which module thread is problematic is equally challenging. In contrast, if it is a multi-process model, whoever’s process consumes a lot of resources should check it; this is still a granularity issue. In the same system, dividing it into multiple processes will result in the complexity of a single process being much lower than that of a single process. Lower complexity also makes it easier to locate and find various issues.Distributed deployment:The internet industry has always emphasized distribution, cloud, etc., and the embedded industry has it tough; it seems there is no need for distribution. In most cases, embedded systems use single chips for independent operation, and distribution is rarely encountered. But what if one day you distribute the functions originally completed by one chip to two chips? Multi-process expansion will be much easier.This is just a special example; in fact, the embedded device industry is a distributed industry; it just started with separation rather than developing from centralized to distributed.Facilitating code permission isolation in companies: In fact, I disdain this practice; companies should trust their employees, but given the lack of integrity in China…, doing some isolation is understandable.Under a multi-thread model, as mentioned earlier, if you remove a module, you might not even be able to build, so should all code be exposed to all engineers? Obviously not, so each module can only provide libraries. However, I think organizing common functional interfaces into common libraries is a normal practice, while providing business-related modules as libraries is a bit…Thus, to supplement, all the advantages mentioned above are not very critical points; they cannot give multi-process models absolute advantages over multi-thread models. They are just personal perspectives that the multi-process model can force engineers to think about solving some problems. (And experienced engineers will consider these issues regardless of the model).Having said so much, it’s time to consider changing the previous project example into a multi-process model; otherwise, it’s just theoretical discussion. Now let’s start:The first problem to tackle is choosing the communication method for multi-processes; direct calls between threads cannot be used, so how to choose the communication method for multi-processes?Linux provides many IPC methods, which I won’t list one by one. For control with non-large data volumes, a better way is to use sockets, and on the local machine, the Unix socket method is more commonly used (What are the benefits of this method? When you need to turn a single system into a distributed system, the advantages become clear).However, simply using sockets to implement the functionality of the previous example will also encounter some problems:Still referring to the previous example, it’s important to note that the second plan we optimized earlier in the multi-process model can no longer be continued. The reason is quite simple; it should be self-explanatory…A simple approach is to base it on the first plan, changing direct calls to socket communication (just define the communication protocol), but engineers familiar with socket development know that establishing socket communication requires some preliminary work (mainly connecting to associate the two modules). Therefore, the previous example will look like this: module a needs to establish connections with modules b and c, and if module f is added, module a also needs to establish a connection with module f. In this situation, if you draw a connection diagram in your mind, you will find that we have woven a spider web; the relationships between nodes are intricate, and similar to the first plan, adding a module related to a requires modifying the code of module a, and this situation is even more cumbersome and complex than the multi-thread model. This approach is absolutely a nightmare.Alright, how to solve this? I think many people will certainly think of using a bus distribution approach. Those familiar with Android system development will think of Binder, those familiar with OpenWRT will think of Ubus, and desktop developers will think of Dbus. Developers in the internet industry certainly also know about the pub/sub module provided by Redis.The principles of Binder, Ubus, etc. are quite simple: establish a message center and build a forwarding routing model. All other modules do not directly interact but use the message center for forwarding and routing, and how to determine the routing rules is defined using the observer pattern of subscription/publishing. (Embedded developers or C language developers often mistakenly believe that design patterns are associated only with object-oriented languages, which is unique to them. Although many experts have popularized this aspect, due to some developers’ closed information channels, this idea still prevails).Based on this model, the requirements of our previous example can be well solved. Adding a message center module allows all modules that need communication to only connect to this message center module, and then subscribe to events they are interested in. When an event occurs, they only need to handle it accordingly.Thus, modules b and c subscribe to the events of module a. When module a detects a certain event, it publishes that event, which first reaches the message center and is then forwarded to modules b and c. For the new module f, it only needs to subscribe to that module and does not need to modify the code of module a, making the expansion of functionality very convenient.At the same time, the previously mentioned custom development is also simplified. If the customized version needs to add module g, this only requires starting module g as an independent process in the customized version and subscribing to the events of module a. The difference between the customized version and the general version lies in whether the process of module g is started, thus achieving a goal of software engineering: adding functionality is like building blocks; you only need to insert (start) or remove (not start) a module, and changes in functionality are limited to one or a few modules, without affecting the main framework.The above roughly describes the gradual optimization process of a project’s requirements. The example seems to be based on an embedded project, but it appears applicable to software engineering as well.Coming to the internet industry:Checking out articles shared by architects of major websites on the technical architecture changes of their sites, the first thing mentioned is generally based on business to split the functions of a previous application server, becoming more refined (for example, e-commerce splitting services for login, registration, transactions, products, sellers, etc.), and then deploying the split services on multiple servers to provide concurrency. Does this sound familiar? Is it similar to the division between multi-threading and multi-processing mentioned earlier?After splitting, the communication issue also arises. Many message middleware has emerged, such as Alibaba’s Dubbo. Simply understanding the principles of these middleware reveals that they are mainly based on subscription and publishing, RPC, etc. mechanisms, which can be said to be similar, while the difficulty lies in protocol formulation and performance enhancement.In contrast to the load balancing solutions in the internet industry, it seems that the load balancer frontend is also like a message center.I have said so much just to illustrate one issue: software design is interconnected, and the underlying thoughts are the same. Although the business logic of the embedded industry is relatively simple, careful consideration will still yield many architectural improvements and designs.However, it saddens me that some embedded developers, given the simplicity of business logic, feel that using some less optimal handling methods can also solve the problem, without considering how to optimize and improve.For example, in the first plan of the above example, if there are not many custom requirements, maintenance might not be a significant issue. Even if there are many custom requirements, hiring a few junior programmers can also manage it; there are companies where one person is responsible for a set of code for a project.Similarly, there should not be an insurmountable wall between the internet industry and the embedded industry; we should focus more on the universal software engineering principles.Disclaimer: This article’s material comes from the internet, and the copyright belongs to the original author. If there are any copyright issues, please contact me for deletion.———— END————● Column “Embedded Tools”● Column “Embedded Development”● Column “Keil Tutorial”● Selected Tutorials in Embedded ColumnFollow the public account Reply “Join Group” to join the technical exchange group according to the rules, reply “1024” to see more content.Click “Read Original” to see more shares.
