RK3588 Edge Computing Application: Detection of Walnut Ripeness
Over the past 60 years, despite the discontinuation of neon tubes, many engineers still have a fondness for them. I created a simulated neon tube clock using 8 LCD screens to experience the aesthetic of retro elements and pay homage to classic designs from the past.
The walnut tree is a woody oilseed species that grows in northern regions, producing high-quality edible oil that is suitable for long-term storage. Currently, traditional methods for detecting walnut ripeness are relatively outdated and suffer from low efficiency in manual detection. The Orange Pi 5 Plus, powered by the RK3588 chip, is a high-performance chip launched by Rockchip Electronics. This chip integrates a powerful CPU and NPU, providing up to 6 TOPS of computing power, making it excellent for complex visual processing tasks, particularly suitable for machine vision edge computing applications. To explore the visual detection potential of the RK3588, we applied it to the agricultural field for walnut ripeness detection. This task involves image recognition targeting walnut ripeness, using image analysis to determine the ripeness status of walnuts, thereby providing valuable information to farmers.
Design Plan
First, we constructed a walnut ripeness detection system, utilizing the computing power of the RK3588 for automated recognition, with detection results shown in Figure 1. The system first captures images of walnuts using a camera, then analyzes the images through a pre-trained deep learning model. The model can identify fruits at different ripeness levels and provide corresponding feedback based on the ripeness grade. When the system identifies that the fruit has reached the optimal harvesting time, it sends a notification to the farmer, indicating that the fruit can be harvested.
This task requires the system to recognize walnuts from camera images and accurately determine their ripeness. Since a single image may contain multiple walnuts at different ripeness stages, and each walnut may vary in size, shape, and color, this task integrates image classification and localization. It is necessary not only to identify the walnuts in the image but also to provide the specific location and ripeness of each walnut, which is much more complex than a simple classification task. The Yolo algorithm is an innovative object detection method characterized by one-time detection, a unified framework, and real-time processing capabilities. Therefore, we chose the newer Yolov8 algorithm.
Hardware List
The hardware list used in this article is shown in the appendix.
Production Process
This system uses a high-definition camera to capture samples of walnuts, which are then transmitted to the Orange Pi 5 Plus edge computing platform via USB. On the Orange Pi 5 Plus, the collected walnut image data is imported into the deployed Yolov8 model, and after rapid training inference by the model, the inference results are displayed on the screen. The system flow is shown in Figure 2. You can scan the QR code on the directory page to obtain the program files.
Step 1: Sample collection of walnuts, with more than 1000 images taken in the field. It is recommended to use the same mobile phone, shooting environment, and shooting angle as in actual shooting, and to perform preliminary screening of the photos to obtain images of fruits at different stages, as shown in Figure 3.
Step 2: Annotate the collected walnut data, using Labelimg for dataset annotation.
(1) Install the Labelimg annotation software using the pip command.
(2) Configure predefined_classes.txt to improve annotation speed.
(3) Launch the software by entering Labelimg in the terminal, click “Open Dir” to open the folder containing the photos, click “ChangeSave Dir” to set the save path to the original path of the photos, and in the View menu, change to auto-save mode (Auto Save Mode).
(4) We categorize walnut photos into 3 classes: Immature for unripe, Ripe for ripe, and Bursts for harvestable. Press the W key to enable the annotation function, draw a rectangular box around the fruit with the mouse, release the mouse, and select the pre-set 3 class labels from the pop-up menu. Repeat the above steps until all images are annotated, forming the walnut dataset, as shown in Figure 4.
Step 3: Configure the Yolov8 environment to train the model.
(1) Install pytorch, ultralytics, etc.
(2) Split the dataset by running the script Split_Dataset.py, dividing the walnut dataset into training and validation sets in an 80:20 ratio, where the training set is used to train the network parameters of the Yolov8 network, and the validation set is used to verify whether the trained parameter weights meet the quantification requirements for walnut ripeness.
(3) Write the configuration file train-config.yaml and the training script Yolo-train.py. Then run the command python Yolo-train.py to train, waiting for the training to finish. The Yolov8 model training is complete, as shown in Figure 5, which displays the training results for the 3 walnut labels.
Step 4: Convert to RKNN model for deployment on RK3588.
(1) Export the best.pt file as an ONNX universal model file, as shown in Figure 6, which shows the completion of the ONNX model conversion.
(2) Install RKNNToolkit2 in the Ubuntu 20.04 system to prepare the environment for model conversion, and prepare the necessary files.
(3) Use convert.py to convert the ONNX model into an RKNN model for easier deployment.
(4) On the Orange Pi board, use the rknn-toolkit2-lite Python API for model inference by running Yolov8.py. The partial results of the model inference are shown in Figure 7.
Experimental Training
During the dataset preparation process, we need to randomly assign the image files to the train (training set) and val (validation set) folders for model training and validation. First, set the dataset division ratio, with the training set accounting for 80% and the validation set for 20%. Then, traverse all images and corresponding annotation files in the dataset folder, and randomly assign them to the training and validation sets according to the set ratio. Finally, move the images and annotation files to the target folders, ensuring that each image and its corresponding annotation file are assigned together. The specific implementation is shown in Program 1.
During the model deployment process, we often need to convert models from one format to another to adapt to different hardware platforms. For example, converting an ONNX format model to RKNN format for running on Rockchip NPU platforms like RK3399Pro and RK1808. Program 2 describes how to convert an ONNX model to an RKNN model and improve the model’s quantization accuracy using the provided dataset.
In practical applications of object detection, we need to load the trained model to perform inference and detection on images. Yolov8.py demonstrates how to load the Yolov8 model and perform inference on specified images, providing a complete process from model loading, image preprocessing, inference, result post-processing, to result display, helping us quickly deploy and validate model performance on different platforms. The main program is simplified as shown in Program 3.
Conclusion
Does deep learning necessarily require a high level of expertise and complex technology? The high-performance edge computing capabilities of the RK3588, along with its wide application and ease of deployment, reduce the difficulty of implementing embedded AI applications. In practical applications, such systems can help agricultural producers make more accurate harvesting decisions, optimize harvest timing, and improve overall yield. If you have an interesting idea based on object detection, after sampling, labeling, and training, the Yolov8 algorithm provides a ready-made training environment and comprehensive training tools, allowing you to take flight with your interests and dreams. This project is funded by the Innovation and Entrepreneurship Training Program of Shenyang Normal University, application number: 202407035.
Related Links
Preview of Radio Issue 3, 2025
Preview of Radio Issue 2, 2025
Preview of Radio Issue 1, 2025
