Today, from university professors to middle and primary school students, from technology workers to ordinary citizens, DeepSeek has become an indispensable intelligent assistant, empowering various industries and driving society towards a new intelligent era. Its emergence is not only a technological innovation but also a profound social transformation.
However, when we use the online version of the large model, we often encounter lagging issues. This is due to the large number of users and high server load, leading to response delays. Additionally, the online large model also poses privacy risks, as user data needs to be uploaded to the cloud for processing. In contrast, local large models exhibit significant advantages. They respond faster and completely eliminate the risk of sensitive information being uploaded to the cloud.
To personally experience the advantages of local large models, we deployed the DeepSeek large model on Yulanben and tested two inference engines based on CPU and RKNPU, respectively, to compare the impact of NPU on inference speed. RKNPU is a Neural Processing Unit launched by Rockchip, designed specifically for edge computing and embedded AI applications. Its core features include high efficiency, low power consumption, and high integration, allowing it to efficiently execute AI inference tasks on resource-constrained devices. For this reason, RKNPU has become an ideal choice in fields such as smart cameras, drones, robots, and smart homes, widely used in scenarios such as real-time facial recognition, target tracking, voice recognition, and visual navigation, providing strong underlying support for intelligent applications.
Next, let’s actually experience NPU inference and see its reliability and resource consumption. My first question was, please calculate the value of 10+5. The AI’s answer was somewhat verbose, but the result was correct, as shown in the image below:
Next, I asked a question that many people are currently confused about: do you think teenagers should learn programming?
The AI’s answer was very comprehensive; it first analyzed the question and then summarized it. Due to the large amount of text, only the final summary part is listed here:
Personally, I believe this answer is very objective, as it neither exaggerates the benefits of programming, encouraging everyone to learn it, nor promotes the “programming is useless” theory, asserting that programmers are outdated. For parents who are still hesitating about whether to let their children learn programming, this may provide a valuable reference.
So how do we know that this inference is using NPU rather than CPU? I ran a top command to monitor CPU usage. When NPU was inferring, the CPU usage was as follows:
Yulanben has 8 CPUs, but when using NPU inference, only two CPUs were used, indicating that most of the processing was done by the NPU.
For comparison, I also tested CPU inference. During CPU inference, the response speed significantly decreased, and the CPU was almost fully loaded, with system resources heavily consumed, leading to a substantial decrease in overall efficiency. The CPU usage was as follows:
It used 7.8 CPUs, which nearly completely consumed the entire CPU of Yulanben.
Regarding these issues, I also consulted the cloud-based large model. When using the cloud-based large model, due to network transmission delays, each question usually takes about 10 seconds to receive a reply. In contrast, the locally deployed DeepSeek can achieve instant responses with almost no delay. Especially during inference, the performance of the NPU far exceeds that of the CPU, not only in inference speed but also with extremely low CPU usage, allowing users to run other programs smoothly without any impact while performing inference.
From the above comparison, it is evident that local inference has a significant advantage in response speed. If you have high requirements for privacy protection and data security, local inference is undoubtedly the best choice.
However, it should be noted that the NPU test used a model with 1.5B parameters (B stands for billion), so when answering some conceptual questions, the richness and depth of its content still lag behind that of the cloud-based large model. However, if the questions are focused on technical fields, the performance of both is almost on par, and local inference can also provide high-quality answers.
-END-
【Shengge Academy】
With sincerity and integrity, we seek knowledge through things.
Viewing software through a humanistic perspective, and changing lives with software technology.
Shengge Academy WeChat Official Account
Shengge Academy Mini Program
Scan the QR code above or search for “Shengge Academy” in WeChat mini program
to read more articles and audiobooks.
Past Recommendations
Shengge Academy GPU Series New Year Opening
Running DeepSeek Large Model Locally
Local Console Output of U-BOOT
Learning from Things: The Search for the Culprit Behind the Black Screen