Everyone loves to talk about edge AI, but they rarely mention the long-standing gap between AI and the embedded world. Edge AI designers are caught in an endless cycle of ‘optimization’, having to tweak neural network models on hardware to achieve acceptable accuracy. They urgently need tools to lighten their burden. This is crucial for the scale of edge AI deployment.Recently, Evan Petridis, CEO of Eta Compute, stated: ‘Today, edge AI is at this unsettling crossroads.’ Edge AI spans two domains, machine learning (ML) and embedded systems. These two distinctly different fields have neither the same language nor the same design philosophy.’The most obvious gap lies in the speed of technology and product development. The pace of ML development is something ordinary hardware designers have never seen before. On the other hand, chips in the embedded field evolve over time, with a much more stable and conservative product development speed.What unsettles the embedded community the most is that experts from the two fields have vastly different views on the rigor of their design and engineering.Neural network models developed by data scientists are statistical. Petridis said, ‘Therefore, when they can get a model to work correctly, say 92%, they think they have won. However, in the traditional embedded world, if you make a mistake, what you deliver will not work at all 100%, and you will face huge economic issues… The operational problems are massive.’Petridis said the result is ‘cultural conflict, domain knowledge conflict, and development cycle conflict.’ He believes this conflict could ‘greatly hinder the deployment of edge AI products.’
Aptos
Eta Compute has just launched a cloud software platform called Aptos, marking another shift in the company’s business model. Eta Compute started in 2015 as an AI chip startup and then became a software IP provider by the end of 2020. Now, it is a SaaS platform company.The transformation of Eta Compute reflects the growing pains of the still-nascent edge AI market and the struggles many AI hardware startups face in finding their ultimate revenue point.Eta Compute explains: ‘Aptos is a new web-based platform that brings an understanding of the embedded systems domain, including the software of the selected chips and the AI capabilities and limitations of the chips.’ The company claims its web-based toolchain ‘can simplify the entire process cycle of model development, deployment, and management for edge ML.’Petridis summarized, ‘We aim to build a tool at the intersection of embedded and ML.’However, Eta Compute is not the first company to develop tools aimed at bridging the embedded and AI domains. Edge Impulse, founded in 2019, claims its toolset ‘makes the process of building, deploying, and scaling embedded ML applications simpler and faster.’Edge Impulse focuses more on ‘the top of the funnel’, making it ‘easy for hardware companies to get started,’ while Petridis believes that Eta Compute’s Aptos will meet the needs of system designers looking to delve deeper and develop ‘production-worthy edge AI models.’The Landscape of the Edge AI MarketThe potential market for the tools from both companies seems vast, as edge AI is a rapidly growing and fiercely contested niche. Of course, AI accelerator startups and AI inference SoC design companies are pinning their hopes on edge technology to gain meaningful shares in the embedded market, as Nvidia has not yet dominated like it has in data centers.Traditional MCU companies like ST, Renesas, and NXP are also working hard to incorporate AI into their product portfolios.For instance, Silicon Labs recently released an embedded IoT platform called Series 3, which includes an AI/ML engine. Silicon Labs CTO Daniel Cooley said, ‘With more memory and computing power, our ML capabilities or vector computing capabilities will improve by 100 times compared to now.’Cooley said that Silicon Labs has comprehensively ‘rethought’ classic MCUs over the past 15 years, stating, ‘These MCUs have been connected from the start, and they will bring more computing power for ML.’ He predicts that ML will ‘become more interesting in the embedded field… just like in data centers, mobile, and automotive domains.’Nevertheless, the actual production rate of edge AI projects is frustratingly low.Eta Compute’s own experience in the edge AI market tells the company that despite numerous experiments, prototypes, and PoCs, edge AI products have not seen mass deployment. This was a major issue for Eta Compute even before Petridis joined, despite having a heterogeneous multi-core SoC optimized for ultra-low power AIoT applications.Deployment delays trouble not only Eta Compute but all AI chip companies.
The Root of Scaling Issues
IDC attributes scaling issues to ‘costs (i.e., hardware accelerators and computing resources), lack of skilled personnel, lack of ML operational tools and technologies, insufficient data volume and quality, as well as trust and governance issues.’
For edge AI developers, the basic scaling issues can be boiled down to three factors: the need to manually craft AI models for accuracy, the constant back-and-forth between data scientists and hardware designers, and the fragmented nature of edge AI applications.
While AI chip companies typically come equipped with their own compilers, they face difficulties in ML optimization. To retain functionality, compilers often downgrade representations, such as from C language to assembly language, or from behavioral RTL to structural RTL.
Steve Roddy, CMO of ML inference IP company Quadric, explains that for ML optimization, compilers cannot do the job because ‘you are actually removing things.’ ‘The toolchain is essentially telling data scientists, hey, you have a lot of excess baggage in your luggage that you really don’t need.’ Roddy likens this predicament to showing up at the airport with overweight luggage, even though you booked a ticket on a budget airline.
Roddy said, ‘Since everything must fit into carry-on luggage, I open your luggage and start throwing out all your clothes.’
This is essentially what happens when data scientists build an extremely complex neural network model. Tasks like pruning, sparsity, or quantization in ML ‘are fundamentally different from what standard compilers do.’
But whose job is it to remove items from my luggage?
Roddy explains, typically it is the embedded personnel and data scientists. Some companies have ‘data engineers’ or ‘ML engineers’ dedicated to filling in the gaps.
Petridis likes the luggage metaphor.
But he adds that the parameters involved in ML optimization go beyond just size reduction and can reshape the model. ‘The multidimensionality of ML can turn your bag into an extremely strange shape, even a star-shaped bag.’
Petridis said, consider a specific neural network operation. ‘You can run it on an Arm core, or if you have an accelerator and NPU, that might help boost the speed by 20 times or 50 times. But it’s not as flexible as a general-purpose CPU.’
Petridis said that given the many variables in hardware and software performance, optimization becomes ‘sparse.’ He noted that it is often difficult for people to keep track of all the variables in process optimization. There are ‘a handful of people, typically from chip companies, who have grown up researching specific architectures and developed a set of heuristics to let you know how to draw things out.’ This approach to optimization may work for a specific project, but if AI chip companies pursue only specific edge AI projects without scaling, it could lead to disaster.
With Aptos, Eta Compute can eliminate the manual sculpting of data science, neural network models, and complex mappings. Petridis explains that it applies ML to discover chip performance, profile chips, and abstract hardware.
When he joined Eta Compute in 2021, ‘we reshaped the company around building the right software infrastructure’ to support edge AI. Petridis explains that Eta Compute is no longer a chip company but has strengthened its ML engineering, with 60% of experts focused on ML and one-third on embedded systems.
Winning the Trust of Embedded Engineers
The Aptos designed by Eta Compute simplifies the back-and-forth optimization process into one step.
Petridis said: ‘We achieve this by abstracting the hardware. We make it accurate enough that embedded engineers can trust it. Ultimately, Aptos provides a model that, for example, achieves 90% accuracy on a specific chip, with a runtime of 17 milliseconds and a power consumption of 1.1 millijoules.’ Petridis emphasizes: ‘This is not an estimate, guess, or approximation. This is a measurement. So if you integrate this model into your software model and run it on your system, you will get exactly the same results.’
Essentially, Eta Compute claims that Aptos can extract maximum capacity from any architecture currently used in embedded AI systems. Petridis said: ‘This is not magic, as our approach is to use the vendor’s tools, diving deep into their compilers because compilers want to allocate tasks in a certain way.’
According to Petridis, the newly launched Aptos is still in the testing phase, with only a few participants in its early access program.
The gap between the ML and embedded communities is wide and deep. An increasing number of suppliers, including Eta Compute and Edge Impulse, believe that the lack of software infrastructure is hindering the potential explosive growth of commercial edge AI products.
from A to B
