Embedded Systems Panorama 6: System Performance and Optimization Practices

Embedded Systems Panorama: System Performance and Optimization Practices

Embedded Linux systems are increasingly applied in fields such as industry, communications, automotive, and smart devices. Performance optimization and system tuning are core aspects that ensure device stability, timely response, and efficient resource utilization. System performance tuning involves not only subsystems like CPU, memory, I/O, storage, and drivers but also includes boot optimization, scheduling strategies, cache optimization, and real-time guarantees. This article will take you from basic principles to practical cases, providing a comprehensive and in-depth analysis of embedded system performance tuning.

1. Overview of Embedded System Performance Tuning

The main goals of embedded system performance optimization are:

1. Reduce boot time

2. Lower CPU usage and improve response speed

3. Optimize memory usage, reduce memory fragmentation and swap overhead

4. Enhance I/O throughput, ensuring storage and network performance

5. Improve system stability and real-time performance

Performance tuning of embedded systems typically follows these steps:

1. Performance Analysis: Use tools to measure bottlenecks in CPU, memory, I/O, boot time, etc.

2. System Trimming: Trim the kernel and driver modules based on requirements

3. Subsystem Optimization: Adjust parameters for CPU scheduling, memory management, and I/O scheduling

4. Driver and Storage Optimization: Optimize driver probe, DMA, cache, and filesystem mount parameters

5. Testing and Iteration: Validate optimization effects through stress testing and tools

This article will combine source code examples and practical experience to comprehensively analyze methods for optimizing embedded system performance.

2. CPU Scheduling and Multicore Optimization

The CPU is the core resource of embedded systems, and proper scheduling can significantly enhance performance.

1. Basics of the Scheduler

The Linux kernel uses CFS (Completely Fair Scheduler) to schedule normal tasks, supporting multicore load balancing; for real-time tasks, it uses SCHED_FIFO and SCHED_RR to ensure priority execution.

Example: View Current Scheduling Policy

chrt -p <pid>   # View task scheduling policy

2. CPU Affinity and Task Binding

By setting CPU affinity, tasks can be bound to specific cores, reducing cache misses and context switching.

cpu_set_t mask;
CPU_ZERO(&mask);
CPU_SET(1, &mask);
sched_setaffinity(0, sizeof(mask), &mask);

3. Multicore Load Optimization

• Load Balancing: The kernel automatically allocates tasks, but critical tasks can manually pin cores

• CPU Isolation: Isolate critical cores to prevent non-critical tasks from preempting

isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

• Effect: Reduced latency for critical tasks, stable system response

4. Interrupt Handling Optimization

• Interrupt Affinity: Bind interrupts to specific cores

• Soft Interrupt and Tasklet Optimization: Reduce the impact of high-frequency interrupts on scheduling

cat /proc/irq/<irq>/smp_affinity
echo 2 > /proc/irq/<irq>/smp_affinity

3. Memory Management Optimization

Memory management is another core aspect of performance, especially in memory-constrained embedded systems.

1. Kernel Memory Allocators

Linux provides SLAB/SLUB memory allocators:

• SLAB: Older version, cache-friendly

• SLUB: Default in modern kernels, efficient with low fragmentation

• kmem_cache_create(): Improves efficiency for frequently allocated objects

2. Page Cache Optimization

Filesystem I/O heavily relies on page cache:

• Use <span>drop_caches</span> to test performance

echo 3 > /proc/sys/vm/drop_caches

• Properly configure cache to reduce frequent write-backs and page faults

3. Memory Pressure and Swap Optimization

• <span>vm.swappiness</span> controls swap usage

• Embedded systems should minimize swap to ensure real-time performance

• NUMA system optimization: Allocate memory close to CPU cores

4. Memory Fragmentation Optimization

• Use slab allocators to reduce fragmentation

• Cache frequently allocated objects

• Avoid large contiguous allocation failures leading to OOM

4. I/O and Storage Performance Optimization

Storage and I/O are common bottlenecks in embedded system performance.

1. Block Device Scheduling Strategies

Scheduler	Features	Usage Scenarios
noop	Simple FIFO	Flash/eMMC
deadline	Limits latency	Real-time I/O
cfq	Fair scheduling	Multitasking environments

• Adjustment Method:

cat /sys/block/mmcblk0/queue/scheduler
echo noop > /sys/block/mmcblk0/queue/scheduler

2. Filesystem Optimization

• Mount Parameters:

noatime,nodiratime,data=writeback

• Filesystem Selection:

• ext4: Stable, suitable for SD/eMMC

• UBIFS: NAND specific, supports wear-leveling

• tmpfs: RAM temporary storage

3. DMA and Cache Alignment

• Use DMA to avoid CPU copying

• Align memory and buffers to 32/64 bytes

• API Example:

dma_alloc_coherent(&pdev->dev, size, &dma_handle, GFP_KERNEL);

4. Storage Performance Analysis Tools

• <span>iostat</span> and <span>blktrace</span> for I/O analysis

• Use <span>ftrace</span> to trace block device scheduling

• NAND testing tool <span>nandtest</span>

5. Driver and Device Performance Optimization

Driver optimization can enhance overall system performance:

1. Probe Optimization

• Delay initialization of non-critical devices

• Avoid blocking in probe

• Use <span>devm_</span> series resource management functions

2. Interrupt and Polling Optimization

• High-frequency devices can use polling mode

• Low-frequency devices should use interrupt triggering

• Reduce context switching to improve CPU utilization

3. DMA and Memory Mapping Optimization

• Prioritize using DMA

• Memory-mapped I/O enhances access speed

• Cache Strategy: write-back / write-combining

4. Driver Debugging and Analysis

• <span>dmesg</span> to view initialization logs

• Use early printk to output critical states

• <span>/sys/class</span> and <span>/proc</span> to check device status

6. Boot Performance Optimization

Boot performance directly affects user experience.

1. Kernel Trimming

• Keep only necessary modules

• Disable unnecessary filesystems and drivers

• Simplify configuration:<span>make menuconfig</span>

2. Bootloader Optimization

• Load kernel and DTB in parallel

• Disable unnecessary initializations

• Serial output is optional

3. Rootfs Optimization

• Compressed filesystem (squashfs)

• tmpfs for caching temporary files

• Delay mounting non-critical services

4. Boot Performance Analysis

• Use <span>bootchart</span> to draw boot time graphs

• Analyze the time taken by each subsystem, focusing on optimizing bottlenecks

7. Real-time Performance Optimization

Real-time performance is crucial for industrial or automotive embedded systems:

1. Real-time Scheduling Strategies: SCHED_FIFO/SCHED_RR

2. CPU Isolation: Critical tasks monopolize cores

3. Kernel Reserved Memory: Avoid OOM

4. Reduce Interrupt Jitter: Bind interrupts, optimize ISR

8. Practical Cases of Performance Optimization

Case 1: Video Processing Embedded Platform

• Problem: High CPU usage, frame drops

• Optimization: Critical tasks SCHED_FIFO, CPU pinning

• Effect: Latency reduced by 30%, stable output at 30fps

Case 2: NAND Flash Write Optimization

• Replaced JFFS2 with UBIFS

• Enabled compression and delayed garbage collection

• Write throughput increased by 40%, lifespan extended by 20%

Case 3: System Boot Acceleration

• Kernel trimming, parallel loading of DTB

• initramfs caching Rootfs

• Boot time reduced from 12s to 6.5s

Case 4: I/O Scheduling Optimization

• Changed eMMC block device scheduler to noop

• Adjusted filesystem mount parameters

• Write latency reduced by 25%, CPU usage decreased by 10%

9. Tools and Debugging Recommendations

1. CPU and Scheduling Analysis: perf, ftrace

2. Memory Analysis: slabtop, vmstat, free

3. I/O Analysis: blktrace, iostat

4. System Boot Analysis: bootchart, dmesg

5. Driver Debugging: early printk, /sys/bus

10. Summary and Methodology

This article summarizes strategies for optimizing embedded system performance:

1. Analysis First: Measure CPU, memory, I/O, boot time

2. Trim the System: Simplify kernel, modules, Rootfs

3. Scheduling Optimization: CPU affinity, multicore load balancing, real-time strategies

4. Memory Optimization: Cache strategies, memory allocator optimization

5. I/O and Driver Optimization: Block device scheduling, DMA, cache alignment

6. Boot Optimization: Bootloader, parallel loading of kernel and Rootfs

7. Iterative Testing: Continuous optimization, combining tools to analyze effects

System performance optimization is a continuous iterative process that requires flexible adjustment of strategies based on hardware, application scenarios, and real-time requirements.

Next Steps Recommendations

After completing the Embedded Systems Panorama series, you have mastered:

1. Linux kernel boot mechanisms

2. Driver model and device tree

3. Filesystem and storage management

4. System performance and optimization

#Embedded Systems Panorama Series #Linux Kernel Architecture #Performance Optimization Practices