Edge AI: Research Background, Key Challenges, and Findings

Research Background and Core Issues

1. The Rise of Edge AI: The Internet of Things (IoT) and AI applications (such as autonomous driving and smart industry) are driving data processing from the cloud to edge devices to reduce latency, enhance privacy, and alleviate bandwidth pressure.

1. Key Challenges: Edge devices (such as Raspberry Pi and Jetson Nano) have limited resources, and their energy efficiency directly impacts the sustainability of AI deployment. Currently, there is a lack of systematic evaluation of device performance and energy consumption, leading to a lack of basis for model and device selection.

1. Research Objectives: To fill the gap by quantifying the performance (F1 score, inference time) and resource consumption (power, memory) of various AI models on different edge devices, providing decision support for edge AI deployment.

Research Methods and Experimental Design

1. Evaluation Framework:

• Unified Measurement Scheme: Combining hardware (USB power meter) and software (psutil time libraries) tools to standardize cross-platform evaluation.

• Metrics: F1 score, inference time, memory usage, power consumption (watts).

1. Experimental Subjects:

• Devices: Raspberry Pi (RPi), Intel Neural Compute Stick (INCS), NVIDIA Jetson Nano (NJn), Google Coral USB (GCU).

• Models:

• Traditional ML: KNN, SVM, Decision Trees (DT), Linear Regression.

• Neural Networks: ANN, CNN, FFNN, R-CNN.

• Deep Learning: ResNet-50, MobileSSD.

• Large Language Models (LLM): TinyBERT, Phi-2-orange.

• Optimization Frameworks: LiteRT (TensorFlow Lite), TensorRT (NVIDIA), OpenVINO (Intel), EdgeTPU (Google Coral).

1. Experimental Process:

• Model Training: Using MNIST (images), ImageNet (object detection), GLUE/OASST1 (NLP) datasets.

• Deployment Optimization: Quantization, pruning, hardware-specific optimizations (such as layer fusion in TensorRT).

• Measurement Steps: Record baseline power consumption → Load data → Execute inference → Collect metrics (average over multiple runs).

Key Findings (RQ1-RQ3)

RQ1: Does dedicated hardware/software enhance edge AI?

• Hardware Impact:

• High-Performance Devices: Jetson Nano (GPU accelerated) and Coral (TPU accelerated) achieve the fastest inference speeds on complex models (such as ResNet-50, LLM), outperforming Raspberry Pi by 3-10 times.

• Energy-Efficient Devices: Raspberry Pi has the lowest power consumption in traditional ML tasks (about 1.5W), but performance drops significantly when handling complex models.

• Software Optimization:

• TensorRT(NVIDIA): Significantly improves inference speed on Jetson Nano (e.g., 40% reduction in inference time for ResNet-50).

• EdgeTPU(Coral): Accelerates TinyBERT on Raspberry Pi, reducing power consumption by 30%.

• Conclusion: Dedicated hardware (such as Jetson Nano) and framework optimizations (such as TensorRT) can significantly enhance performance, but cost and energy efficiency must be balanced.

RQ2: Trade-offs between performance and resources

• F1 Score: All devices have similar F1 scores under the same model (hardware does not affect accuracy).

• Resource Consumption:

• Inference Time: Jetson Nano/Coral << INCS < Raspberry Pi (LLM inference time difference can reach 5 times).

• Memory Usage: LLMs (such as Phi-2) have extremely high memory requirements (>5GB), while Raspberry Pi only supports lightweight models (TinyBERT).

• Power Consumption: Raspberry Pi is the most energy-efficient (for traditional ML), but for complex tasks, Jetson Nano has a better energy efficiency ratio (lower power consumption per unit performance).

• Typical Scenarios:

• Real-time Applications(such as autonomous driving): Jetson Nano + TensorRT (low latency).

• Low Power Scenarios(such as sensors): Raspberry Pi + TinyBERT.

RQ3: How to reduce power consumption through parameter tuning?

• Input Size(visual models):

• When the resolution increases from 224×224 to 512×512, the power consumption of Raspberry Pi doubles, while Jetson Nano only increases by 15%.

• Recommendations: Raspberry Pi should be limited to low resolutions (<300×300).

• Batch Size:

• Jetson Nano supports large batch sizes (16-32), while Raspberry Pi only supports a batch size of 1 (due to memory limitations).

• LLM Parameters:

• Input Length: Raspberry Pi/INCS supports ≤1024 tokens, while Jetson Nano supports 2048 tokens.

• Model Selection: TinyBERT consumes 50% less power than Phi-2, making it more suitable for edge devices.

• Tuning Recommendations: Dynamically adjust input size, batch size, and token window based on device capabilities.

Contributions and Innovations

1. Methodology: The first unified evaluation framework for edge AI across devices and models, with open-source code and datasets.

1. Practical Guidelines:

• Device Selection: Jetson Nano is suitable for complex models, while Raspberry Pi is suitable for lightweight tasks.

• Model Optimization: Quantization + pruning can balance accuracy and resource consumption.

• Parameter Tuning: Input size and batch size are key levers for reducing power consumption.

1. Industry Impact: Provides empirical evidence for the sustainable deployment of edge AI, promoting the development of green AI.

Limitations and Future Directions

1. Limitations:

• Power consumption measurement relies on external hardware (USB power meter), lacking standardized software tools.

• Does not cover more edge devices (such as FPGAs) and models (such as vision Transformers).

1. Future Work:

• Develop automated parameter tuning tools.

• Explore energy efficiency optimization for edge-cloud collaborative inference.

• Extend evaluation to real-time streaming data processing scenarios.

Conclusion

The sustainability of edge AI depends on the triple collaboration of device capabilities, model optimization, and parameter configuration:

• Jetson Nano + TensorRT is the preferred choice for high-performance scenarios.

• Raspberry Pi + TinyBERT is suitable for low-power lightweight applications.

• Dynamically tuning parameters such as input size and batch size can further reduce power consumption by 20-40%. This research provides a data-driven decision framework for the deployment of edge AI, promoting resource-efficient and environmentally friendly AI deployment.