Systematic Methods for Functional Testing of Automotive Embedded Software: Background Knowledge and Methodology

Significant transformations in road traffic systems are related to autonomous driving, electrification, car sharing, and connectivity. In particular, embedded vehicle technology has made significant progress over the past decade. This has led to an increasing complexity in the embedded hardware and software functions in vehicles.

As the complexity of hardware and software continues to rise, the topic of automotive software engineering has garnered widespread attention from scientists and engineers. These developments must consider the integration of other technologies, such as artificial intelligence, computer vision, multi-domain sensors, and multi-agents.

The lines of code in vehicle software have reached hundreds of millions, and this number is expected to continue to grow. With the increasing complexity of interconnected systems that control and manage these vehicles, automakers require new, more complex design tools, particularly those related to software development and testing. Additionally, this leverages automatic code generation technologies, providing a simpler and more practical approach to development and maintenance.

Currently, the functions of software in vehicle systems are diverse and widespread, ranging from low-level control software to high-level and critical safety functions, such as Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicle (AV) functions. All these functions require complex electronic architectures distributed across dozens of electronic control units in the vehicle network. Therefore, the development of such systems necessitates the efforts and interactions of many teams, sharing responsibility with the overall vehicle development. Standards and tools should also follow this trend to ensure that high-quality, safe, and performing applications can be analyzed, designed, and implemented.

Currently, connected vehicles allow Original Equipment Manufacturers (OEMs) to deepen their relationship with drivers through various infotainment, telematics, and vehicle diagnostic devices. Thus, the next phase is to go beyond connected services (such as vehicle add-ons) to wireless (OTA) updates, defining many aspects of the car itself, such as powertrain functions, vehicle dynamics, and new onboard services. Software-defined vehicles (SDVs) can update vehicle functions via OTA updates, enabling vehicles to continuously adapt to the needs of drivers and fleet operators. Therefore, software-based solutions can change operational and ownership models.

Project management of interconnected embedded systems is a significant challenge, especially in the automotive industry, which simultaneously involves multiple participants in processes or workflows, from OEMs to tier-two and tier-three manufacturers. Moreover, the increase in complex functions must not compromise vehicle safety and should be developed alongside the introduction of safety standards, such as ISO 26262, which provides unified functional safety guidelines for all automotive electronic systems. Thus, all stakeholders should be involved to ensure functional and time-sensitive requirements.

Testing can account for up to half of the total cost of software development, and as software complexity increases, the proportion of testing costs may rise unless more effective testing methods are developed. The primary focus of industrial research is to shorten cycles and reduce development costs while improving software quality and refining the software testing process. The entire electronic vehicle testing cycle has several levels, with the main challenge being to determine a testing strategy that detects most failures early in the project.

The main contribution of this series is a systematic approach to automotive software testing: constructing complex vehicle software; organizing testing plans; creating labels to specify functions, systems, and components; easily tracking automotive testing using labels; facilitating the reuse of automotive software; and defining important factors for testing to reduce the number of tests required.

Software testing and verification constitute a significant portion of development costs. Compared to conventional software testing, testing embedded systems software presents additional challenges due to safety requirements. Furthermore, as the complexity of embedded software increases, the costs and efforts associated with testing naturally rise. In this context, testing strategies must be more efficient to reduce costs and eliminate redundancy.

This necessitates a comprehensive development approach and seamless tool support for model-based development and integration. Researchers also emphasize the contribution of scarce literature to the advancement of future complex automotive systems research and support. Similarly, some researchers discuss the automotive electrical system release process, which is very helpful for the industry, especially regarding testing large configurations.

Studies on testing, verification, and validation in the automotive field indicate that the contributions in this field primarily focus on developing methods for low-level model-based testing and verification. This suggests that finding methods that consider the overall picture of vehicle testing is a challenge.

Researchers have proposed an optimized validation strategy to identify and mitigate gaps and redundancies in automotive software testing. This strategy is based on use case trees (UCTs) for each function to better identify use cases and define an overall testing strategy for each function to enhance testing coverage. The UCT method describes the stepwise interactions between systems and subsystems to map the functionalities of embedded systems, which are written in natural language and easy to understand. A model applying UCT is also proposed. This method helps visualize and identify each part of the system, aiding in the elimination of redundancies and gaps in testing.

Another work introduces information on how OEMs organize general testing activities. Most OEMs use the V-model method for system development, where testing levels are divided into several steps, as shown in Figure 1. Functional scenarios can only be viewed to define testing strategies when functional requirements are available. When most specifications are not yet available, a problem arises when planning testing strategies early in the project. The costs of testing activities, which account for a significant portion of development, must be estimated to assess the overall situation of the project and the corresponding testing costs.

Some works introduce multi-level testing strategies, primarily concerning the reuse of testing scripts. These researchers also report on testing from different perspectives, such as integration, bottom-up, and top-down. The discussions extend beyond the details of testing levels, proposing methods to define testing strategies to reduce workloads and share common testing artifacts across various levels. It has also been discussed that the nature of functional testing exists at various levels, and tests defined at higher levels can be reused at lower levels, including adapters for inputs and outputs, as well as parameters suitable for routing at other levels, as shown in Figure 2.

When creating automotive embedded systems, ensuring that the software is complete, safe, efficient, and of high quality is crucial. Therefore, when considering software as a black box, there are three testing approaches. They are:

a) Black Box Testing: It does not contain any knowledge about its internal structure or workings, only checking the functionality of the application;

b) Gray Box Testing: It contains some knowledge that can be compiled to test the application’s functionality and operation;

c) White Box Testing: It contains a comprehensive understanding of the application and its internal structure and processes, not just its functionality.

Static and dynamic testing should also be considered. Test cases should be defined based on the knowledge of the functional architect or developer to avoid mistakes. It is recommended to use tools and methods to automatically construct test cases, such as model-based testing.

Any discussion on automotive electronic system software testing must include ISO 26262. It is the standard for functional safety of electrical and electronic systems in mass-produced vehicles and has specific requirements for software testing. For functional systems with Automotive Safety Integrity Levels (ASIL) A, B, C, or D, certain defined testing requirements must be met, including testing levels and the platforms on which tests are conducted. However, some studies do not consider ISO 26262 when defining testing plans. This topic has been incorporated into the methodology of this series.

We recommend using parts of the method proposed in the references (reference 1, add (Yuan Dongdong WeChat: AlphaApe) for free materials) that indicate that system-level specifications are a common part verified across all tests. Additionally, the authors elaborate on effective validation strategies at lower levels and study each component’s contribution to each function to define testing strategy methods and provide traceability for overall testing. The combination of these strategies, that is, system vision and functional traceability, has been used as a reference to adapt a forward-looking perspective, increasing the assessment of important elements in the automotive industry, including relevant functional safety issues of ISO 26262. This work proposes a method to design a systematic approach for planning and constructing software testing activities in automotive embedded systems.

The automotive industry has adopted the standard V-model for the engineering processes of software-based system development, with a strong focus on embedded systems. The standard V-model is primarily used to describe the testing process as it seamlessly links to the development phases. The authors note that it constitutes a reference lifecycle model for embedded system development and testing in the automotive field. Automotive systems software testing typically occurs at several levels on the right side of the V-model. The exact testing levels of the V-model may vary slightly between projects within automotive organizations and between different organizations.

In this series, the V-model focuses more on the high-level testing activities on the right side, moving upwards from the component level. It consists of four testing levels, starting from the component testing at the bottom to acceptance testing at the top, as shown in Figure 3. The two intermediate testing levels are system testing (representing component packages related to the system) and vehicle system level (representing vehicle system packages).

Due to the diverse types of applications for embedded systems, it is challenging to have a single testing classification that captures all possible systems. However, the authors compile a set of tests based on the results, as shown in Figure 4.

A. Testing Objectives

During the development of products, the testing phase is divided into two main key activities—verification and confirmation processes. The objective of the verification process is to ensure that the product design is correct, while the confirmation process aims to ensure that customer expectations are considered. Verification checks whether the product meets customer needs, while confirmation checks whether the product works as specified (formally or informally). Verification checks whether the system, software, or hardware (or all) meets the requirements, while confirmation checks the coupling of user needs and expectations as recommended by the IEEE 1012 standard. In academia, formal methods for verification are used in software engineering, but practitioners (especially in the automotive industry) do not widely use them.

Verification and confirmation software testing have different objectives. They can be categorized as static testing (also known as review) or dynamic testing, the latter being based on test execution, further distinguishing structural, functional, or non-functional testing. After the review phase, the testing objective is typically to check the functional behavior of the system, while non-functional testing occurs in later stages.

• Static Testing—Testing is generally defined as the process of finding errors, failures, and faults. It identifies the source of code errors without execution. Some examples include revisions of requirements, models, or the test specifications themselves. In contrast, dynamic testing is based on execution. In the automotive field, these testing executions are below the component level, thus not within the scope of this work.

• Structural Testing—These cover the structure of the software under test (SUT) during test execution, such as control or data flow. To achieve this, knowledge of the internal structure of the system (e.g., code or model) is required. Thus, structural testing is also known as white-box testing or glass-box testing.

• Functional Testing—These relate to assessing the functional behavior of the SUT concerning functional requirements. Unlike structural testing, they do not require any knowledge of the internal structure of the software system. Therefore, they are referred to as black-box testing. Functional safety testing is also included in this category. Their objective is to determine the functional safety of the software product and require systematic, executable, and documented procedures. Currently, functional safety testing is only a small part of automotive software testing. With the introduction of safety standards such as IEC 61508 and ISO 26262, the importance of software safety testing will significantly increase in the coming years.

• Non-functional Testing—These are similar to functional testing, but their execution is related to the specifications of system requirements. Unlike pure functional testing, they aim to assess non-functional requirements such as reliability, load, and performance and are typically black-box testing. However, to retrieve certain information (e.g., internal clock), internal access during test execution is required. For example, during robustness testing, the system is tested with invalid input data beyond the range to check whether the system remains safe and functions normally. Robustness is typically ensured through dedicated sanity checks integrated into automotive software.

The contributions in this field mainly relate to low-level model-based testing and verification methods. This work focuses on testing at the highest level, starting from component level (software and hardware integration) until reaching system level.

B. Testing Platforms

Test execution is managed by what is known as a testing platform, which inputs stimuli to the test object and observes and analyzes the outputs of the system. In the automotive field, the testing platform is a vehicle equipped with a test driver. The test driver determines the inputs to the SUT based on driving scenarios and observes the vehicle’s responses, supported by special diagnostic and measurement hardware/software that records data during the test drive and allows offline analysis of behavior. The appropriate testing platform must be selected based on the test object, purpose, or environment.

• Model in the Loop (MiL)—The first integration level, MiL, is based on a model of the system itself. In this platform, the SUT is a functional model or implementation model tested in an open-loop (i.e., initially with no physical model) or closed-loop (i.e., with no physical hardware). This testing checks the system in a simulated environment during early development stages.

• Software in the Loop (SiL)—During SiL, the SUT is tested in either a closed-loop or open-loop configuration. The tested software components are typically implemented in C or C++. However, various high-level programming languages (such as Python and JavaScript) can be used for microcontrollers and embedded systems. The code is handwritten or generated by an automatic code generator based on implementation models. In this series, our primary focus is functional testing.

• Processor in the Loop (PiL)—In PiL, the embedded controller is integrated with proprietary hardware (i.e., ECU) into the embedded device. PiL-level testing is similar to SiL-level testing, but the embedded software runs on-board or on a target processor simulator with the target processor. PiL-level testing is important as it can reveal faults caused by target compilers or processor architectures. This is the last integration level, allowing for debugging during testing in a cost-effective and manageable way.

• Hardware in the Loop (HiL)—When testing embedded systems at the HiL level, the software runs on the final ECU. However, the environment surrounding the ECU remains a virtual structure. The ECU and the environment interact through the ECU’s digital and analog electrical connectors. HiL-level testing aims to reveal faults in the low-level services and I/O services of the ECU. Additionally, final testing of components delivered by suppliers is performed at the HiL level, as the components themselves are integrated ECUs. HiL testing requires real-time behavior of the environmental model to ensure that communication with the ECU is the same as in actual applications. HiL can be conducted only for a single ECU (subsystem), or it can interact with many ECUs in one or more systems, or a complete HiL can simulate an entire vehicle.

• Vehicle—Finally, the last integration level is the vehicle itself. The final ECU runs on a real vehicle, which may be a prototype or pre-series unit from the production line. However, these tests are only conducted in later development stages, are costly, and do not allow arbitrary changes to configuration parameters. Hardware failures are difficult to trigger, and the SUT’s responses are often challenging to observe since internal signals are no longer accessible.

Due to the complexity of automotive systems and the many different characteristics of each function, there are many possibilities to test vehicle functions in different combinations of testing platforms. The goal should be to choose the most effective solutions to balance cost, time, and quality.

Rapid Control Prototyping (RCP) is a very effective method for quickly developing, optimizing, and testing new control strategies in real environments without manual programming. Control software is designed and embedded on RCP hardware. The developed software can be tested in real-time mode, controlling or managing inputs and outputs (I/O) of another real-time hardware connected to the hosted physical plant. Controllers prototyped using real-time simulators are more flexible, faster to implement, and easier to debug. Adjustments or complete modifications can be made instantly. Furthermore, RCP models provide a framework for automatically developing and testing functionalities and features in runtime environments.

C. Testing Levels or Testing Scope

As mentioned at the beginning of this section, testing levels are divided according to the V-model, where steps correspond to project specification steps. During testing, each function may exhibit different types of failures at each level. In Figure 3, the testing levels are divided into component, system, vehicle, and acceptance, focusing on automotive applications. Figure 4 shows testing divided into component and system integration, typically applied to embedded systems.

• Component Level—The component testing level represents the complete testing of a single ECU, where hardware is integrated with software, and the ECU is tested as a black box, focusing on its I/O interfaces and data communication buses, primarily the CAN network, which is one of the main protocols used in the automotive industry. Considering that most ECUs are developed by suppliers rather than OEMs, this verification is likely to be performed on the supplier’s side. Suppliers do not need to request OEMs to retest the component, but must define a test case, work within the agreed-upon test cases with the supplier, and set standard test reports to receive results and maintain traceability.

• System Level—At this level, all ECUs related to the system have been integrated. Prior to this, each ECU was integrated stepwise. All interfaces are tested to ensure interoperability of ECUs. Thus, system-level functional test cases can be used to verify whether the required system functions have been correctly implemented. The primary goal at this level is to test the entire system, especially focusing on the interactions of the ECUs. At this point, many vehicle functions can be fully tested and approved unless testing with real vehicles is required. For OEMs, this level is crucial for integrating supplier components. In some cases, vehicle testing is mandatory and requires the same testing procedures and contributions as system testing. In such cases, the system testing level can be canceled, and directly applied to vehicle-level testing, thereby reducing the overall workload.

• Vehicle Level—When aiming for certification through whole vehicle testing, the vehicle testing level must consider specific elements. There are generally two ways to perform vehicle-level testing: HiL in the lab or real vehicles. The standard ISO 26262 recommends testing relevant safety functions in real vehicles, thus enforcing testing under more realistic conditions.

• Acceptance Testing—Acceptance testing aims to obtain approval from customers and users. Testing primarily focuses on user interface software, known as Human-Machine Interface (HMI). Automotive software testing literature typically does not focus on this level. However, with the increasing number of electronic functions in vehicles, acceptance testing plays an important role in final certification.