25 Lectures on Bus: A Fully Automated OTA Testing Method

Absolute Content

OTA is the star of the automotive electronics field, there is hardly any technical direction more popular and imaginative than it.

Itis both useful and impressive, a typical business that has both substance and appearance, can serve both as a façade and a solid platform.

Excellent companiescan charge for value-added services using OTA, like Tesla.

Tesla officially stated that through OTA upgrades, the rear seat heating function of the owner’s car can be activated remotely, thus enhancing the configuration experience, but the owner needs to pay 2400 yuan for the feature activation. So the question arises, when users purchase a car, the hardware facilities are all installed in the car, but users actually do not know that there is such a function in the car. If it weren’t for the OTA upgrade, users might never know that there is a seat heating function.

Companies that are a little worse can use OTA to patch bugs, covering up major quality defects of their products.

However, when upgrading, the new messages pushed must be downplayed, such as “optimizing vehicle machine smoothness”, etc., to prevent customer panic or distrust, avoiding a brand reputation crisis.

For example, WM Motor!

Back then, this company claimed a range of 400 kilometers when selling cars, but due to technical shortcomings, there were multiple battery fires within a year, and later through OTA upgrades, it was forced to reduce the discharge depth, shrinking the range to below 300 kilometers.

Companies with worse conscience will use OTA for targeted scrapping, extending their reach, making you spend more money at the dealership!

For instance, just after the warranty period, they control the tire pressure sensor to send signals more frequently, draining the battery, forcing you to spend a thousand or so at the dealership!

For example, just after the warranty period, they reduce your seat heating performance, worsen your air conditioning effect, or even reduce your car’s power!

This is much more severe than Toyota’s planned obsolescence, a sure way of precision sales.

For example… (shh, this can’t be said randomly)

Principle of OTA Technology

For specific details, you can search on Google; this article only provides a brief introduction.

As shown in the figure above, this is the most classic and common OTA system architecture.

Taking AiLab’s OTA as an example, users log into the OTA server through a browser, deploy vehicle information, write scripts, upgrade strategies, and upgrade tasks on the OTA server, and push the task to specific VINs.

Immediately, the vehicle machine receives the task and downloads the S19 file (including but not limited to) to the vehicle machine.

When the user starts the vehicle, the vehicle machine will pop up an upgrade prompt, reminding the user to schedule the upgrade.

When the scheduled time arrives, the vehicle machine will first wake up from sleep and activate the entire vehicle network, completing the entire writing process.

Note

For safety reasons, car manufacturers generally do not allow the vehicle machine to connect to multiple CAN lines, but only connect to ICAN, and the writing messages are relayed by the gateway.

The vehicle machine will feedback the writing results, such as software version information, back to the OTA cloud, which users can view through a browser.

Feasibility of Automated Testing

Browser Automation

During the user browser operation stage, the OTA server’s API can be called using Python to achieve automatic task issuance, this part is very simple, so I won’t elaborate..

Vehicle Machine Automatic Appointment

According to the working principle described above, the vehicle machine needs the owner to self-schedule (companies that do secret upgrades are not discussed here).

Regarding the operation of the vehicle machine, we can use adb commands to achieve full automation, and it’s not difficult..

It seems that doing full automated testing on OTA is completely feasible.

After that, we can use the TC1016 CAN card from Tongxing to synchronously monitor four CAN lines, perfectly achieving full automated testing of the entire OTA process..

Common Issues in the Testing Process

BootLoader Writing Test

The writing test is a prerequisite for OTA testing, it tests the reliability of the tested item, simulating the diagnostic tool;

OTA testing, strictly speaking, should simulate the ECU, conducting overall packaged testing of “OTA cloud + vehicle machine + gateway”, as shown in the circled part in the diagram below:

OTA testing schemes cannot be fooled this time

However, given that developing simulation models for each ECU is quite a workload, which has a certain operational difficulty, it is also acceptable to directly include the items to be written via OTA for data recording and analysis testing.

Moreover, if you have already conducted sufficient BootLoader writing tests for each ECU, the ECU is qualified and reliable, and there is indeed no need to create theoretical simulation models, the “data recording analysis” method of OTA testing is also acceptable.

Network Management Testing

OTA is generally conducted in the early morning, when vehicles are usually in a network sleep state, thus, the network management and low power performance of the ECU are also part of the OTA business testing.

What we hope is that the ECU can wake up successfully and reliably, can sleep without delay, and the current consumption drops to a very low level. For this part, you can refer to another article on vehicle technology:

Lecture 53 on Bus, AutoSAR Network Management Automation Testing System Based on Python or LabVIEW [Detailed Explanation]

Positioning of OTA Testing

Try to make it a bench test, as bench testing can simulate enough abnormal conditions, allowing for sufficient testing of OTA performance and reliability.

Real vehicle testing should minimize simulated conditions, only testing “can it upgrade”.

Real vehicle testing can be arranged for outsourced personnel, and clearly state the requirements for testing record files, such as vehicle information, OTA task details, CAN network logs, vehicle anomaly videos, etc.

Gateway Frame Loss

As a CAN product, not losing frames is the industry bottom line, but we indeed sometimes encounter awkward suppliers who do not even care about the bottom line.

Sigh!

Frame loss can lead to writing failures, this is beyond doubt.

In response to this, we need to conduct special forwarding performance tests on the gateway, raising the load rate to over 80%, sending a large number of messages, and the data of each message should preferably not be the same, to see if it really loses frames and when it will lose.

Sending messages is also very simple, using the TC1016 CAN card from Tongxing, it can be easily achieved.

However, for extremely high loads, it seems that currently it can only be achieved through the code editor that comes with Tongxing, and cannot be done like the Zhou Liguang CAN card where messages are sent in bulk through dll…

We believe that Tongxing will soon optimize this and release a new version of the API, we look forward to it!

Countermeasures

If it is confirmed that the gateway will forward frame loss, what to do?

It should be known that a casual s19 file has hundreds of packages, which makes it very easy to encounter writing failures.

Currently, there are two methods for this:

1 is for the vehicle machine to perform a large loop, exit the current writing session, and restart the writing process. This method is simple, does not require improvements to the existing ECU logic, but is time-consuming, and if frame loss is severe, it may never be written in.

Long time consumption significantly increases the risk of OTA causing battery depletion.

Especially for secret upgrades, the owner may feel that someone is tampering with their car at night, and may even call the police, and if some media expose this, then the reputation is completely ruined.

2 is to resend the package if frame loss occurs during the transmission of a specific package.

If the last message of this package is lost, the vehicle machine will not receive a response from the ECU at the P2 Server, and will resend the package, up to 3 times; if it still does not respond after 3 times, it will exit this writing session; if it is not the last package that is lost, but a message in between, it is equivalent to “jumping the sequence number”, and the ECU should identify this and immediately return 7F, allowing the vehicle machine to resend the current package faster.

However, in any case, it is required that the ECU can accept “package resends”, even if it is a resend that is unnecessary, the ECU should not report an error.

Clearly, the second solution can solve the problem once and for all, and the reliability of the ECU is higher, with greater fault tolerance.

Signal Simulation

OTA upgrade activities have signal requirements.

The vehicle machine itself will recognize and will not execute the upgrade activity if the conditions are not met.

The writing process will also recognize it; there is generally a step called “check writing conditions”, the ECU will check the bus signals, and if they do not meet its prerequisite definitions, it will respond with “not meeting writing conditions”.

If the test bench only includes the vehicle machine, gateway, and tested ECU, it will be simpler, as you can freely simulate CAN signals.

If the test bench connects many ECUs, it becomes a bit troublesome, as these signals are all real, and you cannot just send them randomly.

There is a way to do this; you can conditionally intercept and forward the CAN line of the ECU, altering the signal values.

Regarding this, CANoe can easily achieve this, and the performance is good; we have also previously detailed the tutorial on calling CANoe with LabVIEW.

Lecture 22 on Bus, Tutorial on Calling CANoe with LabVIEW, generously offered

Tongxing’s mini-program can also easily implement this, and the performance is also good, and it can also be achieved through LabVIEW calls.

Format for Test Reports and Test Cases

Shizi No. 1 strongly recommends Excel; the Excel format for use cases and reports is a mark of self-respect for engineers.

In the early stages of project development, we can spend some time debugging such a testing system, preparing test cases in advance, and as soon as the software arrives, we can start testing immediately.

Automated testing has inherent advantages in generating reports and backing up data; the documents are exceptionally clear, the LOG naming is meticulous, and even the timestamps are in milliseconds, with the Excel background color being rich in graphics and text, which makes one marvel at the limitations of manual work.

Leaders love it, reports are very effective!

You can record many things in the Excel report, such as:

1. Detailed parameters of the OTA cloud task, such as task name, vehicle VIN, pre-writing software version of the ECU to be written, target software version of the ECU to be written, version of the ECU to be written displayed on the cloud, limiting conditions (voltage, vehicle speed, gear, door status, etc.), task push time, silent or scheduled…

2. The reminder time for “vehicle machine upgrade available”. You can subtract this from the task push time to calculate the vehicle machine’s download performance.

3. The actual version of the ECU to be written on the vehicle side. The vehicle machine will read the ECU version before upgrading; you can listen to their CAN dialogue, decode and record it, and write it into the report. If it does not match the “version displayed on the cloud for the ECU to be written”, that is also a problem.

4. Bus load rate, diagnostic response time, single package sending time and its normal distribution statistical analysis

5. Complete record and analysis of the BootLoader writing process

6. Whether the command returns when communication is turned off, whether communication is actually turned off; whether the command returns when communication is turned on, whether communication is actually turned on.

7. After writing, read the ECU version number again and monitor the version number displayed on the cloud in real-time through the API, calculating the timeliness of ECU information upload and recording it.

8. Key logical behaviors on the vehicle side. Such as the time when the vehicle machine sends the first frame wake-up message, BCM power-on signal, IVI_OTA control signal, last frame network management message time, vehicle network sleep time, vehicle no-ack time, etc.

9. Records of special operations. Such as simulating the pressing of the BCM start button and its time, simulating the pressing of the key button and its time, controlling the program-controlled power supply voltage and its time, etc.

The above are just part of the data analysis capabilities of the OTA automated testing system; in fact, you can significantly add more, as long as you know the logic and behavior, you can monitor all of them.

Moreover, this is a one-time process; once added, it will be useful forever, regardless of the task load, as it is automated.

Real Vehicle Testing

Real vehicle testing is a practical verification of all our previous work; our ultimate goal is still to perform excellently in the car!

Real vehicle testing is also a supplement to the above OTA automated testing; some anomalies are unimaginable and need to be improved in practice, thus optimizing the automated testing system itself.

Especially in the small batch trial upgrade stage, statistical analysis becomes particularly important.

Real vehicle testing does not have a specific methodology, so you can only improvise; whatever problems arise, just solve them.

Further Communication

After reading the above content, are you filled with confidence about OTA testing?

So what are you waiting for? Just get started!

The Shizi team can, and so can you!

If you’ve reached this point and still don’t understand, add us as friends in the backend, don’t hesitate, we’ll teach you hand-in-hand, sharing the best practices in the industry with you.

No need for millions, really not necessary.

【Recommendation】

In the automotive industry, never go for outsourcing

Hengrun and Yian, two windows to understand the past, present, and future of the industry

Lecture 10 on Bus, the best Excel2DBC tool in the Eastern Hemisphere, always free to give away