Systematic Approach to Functional Testing of Automotive Embedded Software (Part II): Structure and Planning of Software Testing

As electronic systems become increasingly complex, the first recommendation for this work is to propose a method based on the V-model to construct interfaces for functionalities, components, and systems, facilitating the definition of activities and responsibilities.

The general electrical system of the vehicle can be defined as a set of systems, each responsible for grouping a set of functionalities related to its subject, with multiple components responsible for executing functionalities, as shown in Figure 6.

Analyzing the V-model in Figure 3 from series “I”, all testing levels are directly related to the project specification steps. Based on this, a testing execution strategy is defined, which can identify redundancies and gaps. Considering that the testing plan directly impacts project costs, the testing strategy must be confident and objective from the project’s outset. The four levels are the basis for applying this method to organize and construct the complexity of automotive systems:

1. Functional description;

2. System specifications;

3. Mapping of systems, components, and functionalities;

4. Defining the basic factors affecting software testing in the automotive industry, as shown in Figure 7.

They will be described in the next subsection. Then, the strategies for defining each functional testing level are integrated (Section 5.E). Section 5.F shows that the structuring of testing activities can achieve tracking, management, and control of testing activities.

From the OEM’s perspective, the proposed method uses minimal available information from the vehicle’s electrical/electronic systems to define the strategy for testing software. This is an ideal method that allows testing the project scope early, controlling the entire project testing plan, covering all functionalities, systems, and ECUs of the vehicle. The functional layer will be described in more detail below.

A. Functional Description

The vehicle has multiple functionalities, which are written in a way that is easy for users to understand, especially in the initial stages of the project. In this case, the target users can be passengers, drivers, operators, maintenance personnel, or production workshop staff.

In the initial stages of the project, the description of functionalities is independent of hardware or software. In the industry, this is recorded in natural language, which can be understood by non-professionals. Its description does not consider how the functionality is implemented and its interdependencies within the vehicle. Table 1 shows the proposed model for specifying functionalities. All functionalities of the vehicle should be identified (Fn), named, and given a basic description. As an example, the functionalities used in the references (Reference 1, add (Yuan Dongdong WeChat: AlphaApe) to receive for free) are described.

B. System Specifications

The vehicle contains multiple systems, each bundling a set of functionalities. Different companies define systems differently, usually as a name related to a set of functionalities. Here, it is recommended to use a basic model to describe the system according to Table 2, which contains basic information. Each system will receive an identifier, a name, a brief description of the relationship between the system and specific functionalities, and the last two columns are the names of the functionalities represented by the system.

Each functionality has a system as its owner, even if other systems can also participate in that functionality. For example, the fuel level has contributions from the power system (S3) to measure and check the fuel level failure.

C. Mapping of Systems, Components, and Functionalities

In this case, mapping uses the concept of the architecture of electronic systems, which defines how components and systems are organized. This includes mapping functionalities within the system architecture, presenting each component within its system, and showing the correlation of each functionality between components. The mapping focuses on important data interactions such as vehicle electrical systems, functionalities, systems, components, and contributions of functionalities. The lowest level of information in this method is the specification of functionality contributions, which indicates the part of the functionality that each component is responsible for.

In this step, functionalities are mapped within each component and system, represented by lines connecting each component that participates in executing the functionality. In the example shown in Figure 8, we use the same functionalities reported in the literature. The functional architecture contains three systems: cockpit, powertrain, and external lighting. The cockpit system represents a set of functionalities close to the driver’s position, with components including: dashboard, ignition controller, and headlight switch. The powertrain system represents functionalities related to vehicle propulsion, while the external lighting system represents all functionalities related to the vehicle’s external lights. The green lines connect the components participating in the external automatic lighting functionality, while the blue line connects the fuel level display functionality.

This is a specification step for the system, where each component has its defined list of functionality contributions. Operationally, Table 3 defines the basis for representing this system mapping. In the case of the powertrain system, it helps with the fuel level detection functionality, but the system responsible for displaying the fuel level functionality is the cockpit system.

In addition to Table 3, Table 4 also proposes a matrix to map the entire vehicle system, combining all previously defined information to indicate how many components and systems participate in each functionality. One of the additional points in this table is the functionality contribution ID FnSnCn, which represents a unique code for each functionality contribution and becomes part of the functionality’s “DNA”.

Here, the minimum level of functionality specifications is the functionality contribution and identifier FnSnCn, so that this code can identify the components and systems responsible for that contribution. If a component has multiple contributions to a functionality, Fn will receive a point, and the contribution number can be added after the point. For example, functionality F3 has contributions from components C5, C7, and C9. This method maps functionalities to systems and components, where each contribution has a unique number. This number can be easily tracked, providing a systematic and complete way to visualize vehicle functionalities. This mapping method can support any management activity, not only for testing but also for the complete development, production, and after-sales lifecycle. For example, if certain functionalities fail in vehicles on the field, the specification history, testing, and reporting can be easily tracked. From left to right, we view the layers as top-down from user to component. We can then classify it as system, functionality, and component. To ensure completeness, it is suitable to define testing patterns at all levels.

Considering that the identifier has the functionality’s “DNA”, this method also helps in developing a tool to manage and organize the software development process. This provides a reference for developing a framework to document each functionality, the functionalities connected to components and systems, and their interfaces.

D. Automotive Characteristics

As described in Section 2 of series “I”, the automotive industry has specific characteristics and challenges during product development that must be considered in the testing strategy. Among them, the characteristics of applications, safety, and quality standards can be emphasized, as well as some testing requirements, relevance from the customer’s perspective, complexity of the systems, and complexity of testing execution.

Safety-critical embedded systems have factors that are proposed and can be evaluated here: safety level, different complexities, impact of external environment or operator on system functionalities, and, in some cases, customer involvement. As shown in Table 4.

1) Functional Safety

The main element for defining the testing strategy related to functional safety factors is to conduct a risk analysis and assessment at the beginning of the project lifecycle, where the automotive safety integrity level (ASIL) is determined for each system. Based on this classification (identifying systems with accident risks), requirements to avoid residual risks are defined, and testing methods are recommended to verify and ensure system approval.

Software does not have a direct impact on risk, but at the same time, it can often be seen as responsible for system-level safety, as software can propagate faults through system elements. Understanding the entire ecosystem of application software functionalities is crucial. This information will influence and assist in defining the software testing strategy and is one of the factors in defining the testing plan.

The identification of ASIL for each functionality begins with hazard analysis, where all hazard events are identified within the focus of functional safety. There are several methods in the literature that use the ASIL concept for risk analysis. Each hazard event is classified based on the potential severity of injury (S), and the exposure (E) (the expected relative frequency of working conditions that may cause injury) and control (C) (the relative probability that the controller can take actions to avoid injury) are further classified into combinations to assess the likelihood of injury risk.

The ASIL method recommended by ISO 26262 for risk classification is used to assess each functionality of the vehicle. As shown in Table 4, there are five possible outcomes: QM, A, B, C, and D. When the level is A, B, C, or D, the functionality of the system or component has functional safety relevance. QM level means that the risk related to hazard events is not safety-related and does not require compliance with ISO 26262 safety measures.

Once the specifications for functionalities, systems, and mappings are defined, the ASIL method should be individually applied for risk level assessment for each functionality. This classification is one of the parameters for defining the testing strategy.

ISO 26262 has some recommendations, and Table 4 defines the tests that should be performed from each ASIL level. The ASIL levels are given by the following formula: “++” indicates that the method is strongly recommended for the identified ASIL, “+” indicates that the method is recommended for the identified ASIL, and (*) indicates that the network environment can be partially or fully integrated with the vehicle system testing platform.

According to ISO 26262 recommendations, when ASIL is A, B, C, or D, software must be tested under actual conditions of the vehicle. However, in cases A and B, if component and system-level testing is performed on the testing platform or HiL, vehicle testing can be omitted. In this case, the goal is to provide evidence that testing can be conducted in representative conditions simulating real vehicle scenarios. However, this should be well documented because if an incident occurs in the field, the OEM should be prepared to explain the details of the testing. In our recommendations, to avoid misinterpretations in such a relevant subject, if the functionality is ASIL A, B, C, or D, real testing on the vehicle must be conducted. For this method, regardless of the initial definition of the testing strategy planning, it is important whether the functionality is related to functional safety.

2) Customer Relevance

Customers must be involved in the software testing strategy, bringing collaborative design aspects into the proposed method. In practice, it should be noted that over the past few decades, software has become a part of functionalities and innovations in embedded systems across various fields, from telecommunications and medical devices to automotive systems. Many products that include embedded software have been developed across multiple domains, increasing the relevance of stakeholder involvement in product development (including software validation). These changes have led to a situation where software engineering has become a significant topic in embedded systems, a crucial participant in innovation and market competition. It directly impacts the availability of vehicle functionalities. This factor must be considered in software testing.

Most methods in the literature do not consider customer validation in the testing strategy. This validation is one of the highest levels in the V-model, referred to here as acceptance testing. To plan and define the testing strategy, the necessity of acceptance testing should be evaluated, and a question should be answered in advance: Does this functionality require customer approval before production?

3) System Complexity

Automotive embedded systems are a noteworthy example in terms of components and functionalities. Figure 9 shows that multiple objects must be developed and tested, from the lowest level to the highest level: models, software units, embedded software, hardware and software components, systems, and the vehicle itself. Testing can be performed at different levels, and a systematic approach must be developed to promote understanding and definition of testing.

This work focuses on the highest levels of software testing, thus objects (modules, software units, and integrated software) are not within the context, as in most cases, the complexity of these lower-level tests is lower and usually managed by the suppliers, with final reports submitted to the OEM.

Here, complexity analysis is based on the number of interfaces that each functionality has, how many components and systems participate in that functionality, and how many functionalities each component or system has. Some questions must be raised to construct the definitions of testing at the component, system, and vehicle levels.

4) Driver Influence on the System

There are many types of functionalities in vehicles, ranging from very simple and passive functionalities to active functionalities that influence functional safety. The behavior of some of these functionalities is affected by the driver. If the driver’s behavior cannot be simulated in the laboratory, vehicle testing is required.

To facilitate the application of this factor in the proposed method, two questions are included to assess each functionality:

5) External Environment Influence on the System

New technologies interact more with the external environment, generating multiple variables within the vehicle system, which cannot always be simulated in the laboratory.

Traffic factors, different types of user behaviors and reactions, and more dynamic driving situations, mainly for autonomous vehicles, can be mentioned. Another example is related to connectivity or electric vehicle functionalities, where the city is part of the vehicle ecosystem. For instance, the structure of the Internet of Things (IoT) or the system infrastructure for battery charging, which requires verification in a real environment with the vehicle to obtain final product approval, should be considered in testing planning.

Most new vehicles have powerful networking, communication, and data processing capabilities, enabling communication with other vehicles or exchanging information with the external environment through various protocols (such as Wi-Fi and Bluetooth). Therefore, many innovative telematics services (such as remote safety to disable the engine and remote diagnostics) have been developed to enhance driver safety, efficiency, and comfort. The next step in this evolution is the surge of autonomous vehicle solutions.

In this case, the interfaces of vehicle functionalities often surpass embedded systems, requiring the vehicle to validate software functionalities in a real environment, as simulating in the laboratory is not always feasible. Similarly, for driver influence, some questions must be answered to define the testing strategy:

E. Assessment of Required Tests for Each Functionality

The previous sections introduced important factors influencing the definition of testing plans in the automotive industry, the specification models of functionalities and systems, and how to map functionalities within the system. Figure 10 summarizes the structure of the method proposed in this series of articles.

This section introduces the definition of a multi-level testing strategy based on the information from the previous sections. The first flowchart described in Figure 11 assesses factor 1, which is the relevance of functionalities to functional safety.

This is one of the most important factors in automotive applications, as electronic system failures leading to accidents can have legal and economic consequences. ISO 26262 has specific requirements for functionalities related to functional safety, and the testing plan must follow these recommendations. The method proposed here divides the multi-level testing strategy and plan into two parts: testing strategy for relevant functionalities related to functional safety (ASIL A, B, C, and D) and testing strategy for functionalities not related to functional safety (QM), as shown in Table 6.

1) Process for Defining Testing Strategy for Safety-Related Functionalities

Functionalities related to functional safety should use the flowchart in Figure 12. The first step of the flowchart assesses factor 4 (system complexity). If the functionality interacts with multiple components of the same system, system-level testing must be applied. In cases where the functionality interacts with different systems, functionality is only completed at the vehicle testing level, but there may be functionalities where components have system-level interactions. In such cases, system-level testing is relevant and should be considered; otherwise, system-level testing can be omitted.

The last parameter to assess is factor 2 in the flowchart in Figure 12, which is the customer relevance in functionalities. If relevant, acceptance testing should be considered to plan customer involvement in functionality validation.

2) Process for Defining Testing Strategy for Non-Safety-Related Functionalities

Combining the parameters defined in Section 5.D and using the workflows in Sections 5.E.1 and 5.E.2 can generate eight possible testing scenarios, as shown in Table 7. Vehicle-level testing may be driven by three different parameters: safety, complexity, or external and driver influences. The question marks in the table indicate that if one parameter in the row is set, it is sufficient to define the need for vehicle testing. In summary, the sequence of activities for defining testing levels is:

• Specify functionality;

• Specify system;

• Define system architecture and mapping of functionalities within the vehicle system;

• Fill in the four parameters of the application (Section 5.D);

• Use the flowcharts presented in Sections 1 and 2 to define testing levels;

• Fill the results into the template (Table 7).

When all functionality parameters and defined testing level scenarios are filled in Table 7, it will be easy to display the testing volume view for each component, system, vehicle, and acceptance level, as shown in Figure 13. The testing order starts from components, systems, vehicles, and acceptance. The goal is to conduct as many tests as possible at the beginning of the project to reduce later risks. The proposed method adopts the criteria defined by the classification factors (Table 7), which can facilitate extensive testing at the early stages of the project to improve quality and meet requirements and specifications. Relativizing the level of each item in a general form can assume a certain level of risk for each testing phase. This is useful for project management, for example, investing more funds in simulation techniques to reduce risks and project timelines.

This method provides a systematic approach to organizing and controlling activities, making it easy for even non-professionals to use. This brings greater flexibility and organization to embedded system projects.

F. Testing Traceability

The term “traceability” is specifically used here to manage testing activities and control the alignment of test cases. It is divided into two parts:

a) Identification of testing activities;

b) Alignment of test cases.

1) Identification of Testing Activities

To establish effective traceability, it is important to map functionalities, identifying each testing step and process. In the case of automotive systems, there are usually multiple abstract levels of requirements, and component-level only represents a part of the testing activities of other levels, and its scope and responsibilities must be defined.

Once each functionality defines the testing levels required to approve its functionality, a method must be defined to organize the responsibilities and traceability between testing levels, eliminating gaps and avoiding testing redundancy.

Each testing level has a predefined sequence of steps to execute functional testing, and each level is defined separately without the need for interconnections with other levels or traceability of results between them. Figure 14 shows the various interactions that functionalities can have between components and systems. In this case, two components C1 and C2 in system S1 are connected, and component C3 in system S2 has another part contributing to functionality F1.

By analyzing the testing plan of this example, especially factor 4 (complexity), the following phases of functional testing should be presented:

• First Phase: Component Testing. The individual testing of each component verifies the contribution of functionalities at the component level, with component C1 testing the contribution of functionality F1S1C1, component C2 testing functionality F1S1C2, and component C3 testing functionality F1S1C3.

• Second Phase: System Testing. Testing the combination of functionalities F1S1C1 and F1S1C2, which are part of the same system S1. System-level testing may have two situations: functionalities that do not interact at the system level, meaning no changes in component-level and system-level testing scenarios; or interacting scenarios. For example, it can be a simulated signal at the component level of C1, while the system level directly receives it through C2’s CAN bus.

• Third Phase: Vehicle Testing, in this case only involving S1 and S2, which have a combination of all functionality contributions representing functionality F1.

• Fourth Phase: Customer Acceptance Testing for the functionality.

The mapping defined in Section 3.C of series “I” provides a method to identify each system, component, and functionality contribution using unique codes. Similarly, we have described unique identifiers for each testing phase of functionalities, allowing tracking of activities by level. Given that documentation and history of each activity are necessary, mainly to build the complexity of software testing activities in automotive embedded systems.

Testing levels are identified by two letters: Component Testing (CT); System Testing (ST), Vehicle Testing (VT), and Acceptance Testing (AT).

Each functionality with the identification of testing levels must comply with the characteristics of the tests, as described in Table 8 and Figure 15. At the component level, each activity adds only the letters CT to the functionality contribution code (CTFnSnCn). In system testing, the primary identification of this stage relates to the components of functionality testing. The identification of this step may have multiple components (STFnSnCnCn). At the vehicle level, the focus is on which systems represent the functionality, and the identification of this step may have multiple systems in the composition of the code (VTFnSnSn).

Table 9 shows the code model and the interconnections between activities at each testing level. At the component level, when each component has multiple functionality contributions, the second column uses the functionality contribution CTFnSnCn. The first column defines minor contributions, which are subdivisions of the main part CTFn.SnCn. The index “n” after “Fn” refers to subcomponents.

2) Alignment of Test Cases

Test cases are a series of specific testing steps used to check all aspects of a system or component, including inputs and outputs, detailing the steps that must be considered and the results that must be achieved. Typically, each functional testing level has a specific test case, without alignment with the content of other levels.

In the automotive industry, most components are developed by suppliers. In this case, it is ideal for the OEM to receive components that have been thoroughly tested as black boxes, requiring only integration. However, currently, these components are part of combinations of vehicle functionalities, making it difficult to isolate components; therefore, integration testing is essential. To improve efficiency and utilization at all levels, testing steps need to provide a model, i.e., to utilize the testing of a certain level at a higher level, avoiding testing redundancies or even gaps in testing.

At the component testing level, most of the work is performed by suppliers, reducing complexity in the testing environment due to the focus on component behavior, source code failures, and integration with component hardware. Keeping most tests at the component level can identify failures early, while integration and validation testing of interfaces between components are left for other levels like systems and vehicles. Another important point is that if errors are discovered late in the development process, the cost of correcting them increases dramatically. Figure 16 shows the timeline of testing types in the automotive industry, keeping most tests at the lowest level, which is highly beneficial at the component level.

One of the strategies to optimize the results of each level of testing is to create links between test cases at different testing levels. The recommendation here is to use a multi-level testing model, where the testing sequence at the highest testing level is defined as a common testing core for all levels.

The proposal asserts that a test case for a functionality consists of a common core part, called the Test Case Core (TCC), which is defined as the testing script for the highest level of testing functionality. For the lowest level of testing, some adapters will be added when needed: Output Interface Testing Adapter (OTA), Input Interface Testing Adapter (ITA), and Parameter Testing Adapter (PTA), as shown in Figure 2. TCC is shared across all testing levels, and the high-level testing roadmap is reused. Only the adapters used at each testing level are specific.

TCC describes the testing steps for a given functionality at the system level. TCC is the basis of the testing routine defined by the OEM. Lower levels will reuse this testing routine by adding OTA, ITA, and PTA when needed to customize interfaces and parameters. ITA and OTA must simulate the behavior of components that are unavailable at the given testing level to achieve the functionality expected by TCC. PTA must map any testing parameters applicable to the testing object.

Here, the use of TCC is only applicable to the testing levels of components, systems, and vehicles. In the case of acceptance testing, this concept is not used, as this testing does not have a standardized sequence that can be easily reused at other levels. When acceptance testing is needed, it will receive a specific testing sequence.

As mentioned earlier, most embedded systems in the automotive field consist of components developed by suppliers, but the ultimate responsibility lies with the OEM. The greatest challenge is integrating components into a common electronic vehicle system, where many functionalities interact to achieve other functionalities within the vehicle. Using this approach, functional testing (the focus here) is controlled and aligned at all levels, as shown in Figure 17. The testing sequence at the highest level is designed to serve as a reference for all other levels. This supports the goal of keeping all sequences of core components at lower levels consistent and aligned with the sequences defined at the highest level by the OEM.

This model’s application provides a way to align functional testing across all levels. The code is added to testing activities that reuse test cases from another level, where the components or systems that reuse TCC receive the identifier code RE at the end to identify that activity as consistent with the TCC of that functionality. The code for component testing activities is TCFnSnCnRE, while for systems, it is TSFnS2CnCnRE.

After understanding the structure and planning of software testing in the automotive industry, please stay tuned for the subsequent release of series “III”: Model Application and Conclusion.