Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPieIntegrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Currently, the field of HMI design is facing challenges in the integration of software and hardware, and designers need a tool to break down the barriers between design and engineering. On March 19, 2025, at the 5th China Automotive Human-Machine Interaction and Experience Design Innovation Conference, Li Li, Business Development Manager for the Asia-Pacific region at Studio XID Korea, introduced that ProtoPie is dedicated to creating high-fidelity interactive and animated prototypes across screens without coding, allowing designers to conduct design validation in a realistic environment that combines software and hardware.

Li pointed out that by utilizing ProtoPie’s API plugins and bridge app mechanism, third-party AI services can be flexibly called within HMI prototypes. With built-in plugins and the bridge app, design teams can quickly achieve software and hardware interaction.

Li also shared case studies of ProtoPie in practical applications, showcasing its powerful capabilities in zero-code cross-screen interaction, software and hardware interaction, building desktop setups, and integrating 3D and web elements.

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Li Li | Business Development Manager for the Asia-Pacific region, Studio XID Korea, Inc.

The following is a summary of the speech content:

What can ProtoPie do?

ProtoPie is a zero-code interactive prototyping tool designed for smart devices, applicable to various smart platforms including mobile phones and in-vehicle displays. As an innovative tool, ProtoPie aims to eliminate the gap between design and engineering. In the automotive and smart device fields, new hardware technologies and AI applications are constantly emerging, presenting designers with the challenge of integrating these cutting-edge technologies into interaction design, and ProtoPie was created to address this issue.

The core functionalities of ProtoPie are reflected in several aspects: firstly, it allows designers to create high-fidelity interactive and dynamic prototypes across screens without writing code; secondly, ProtoPie provides a platform that simulates a real environment, enabling designers to validate interaction designs that combine software and hardware, significantly reducing reliance on development resources; thirdly, through ProtoPie, design solutions can be delivered to development teams with one click, avoiding cumbersome documentation and explanation processes, thus improving work efficiency.

ProtoPie not only supports zero-code integration of hardware and HMI design but also seamlessly integrates cutting-edge AI technologies, such as the highly regarded DeepSeek and ChatGPT, into HMI design. This functionality empowers designers to explore and implement AI-driven interaction innovations without complex programming.

The core of achieving the above functionalities lies in the ecosystem built by ProtoPie, which consists of four key components. First is ProtoPie Studio, the core client for designers to create interactive prototypes, where designers can freely create and edit interactive prototypes. Second is ProtoPie Player, an application for mobile devices, especially for Android devices and Android car screens, that allows for quick demonstration and testing of prototypes, enabling users to easily install this application to view and test prototype effects instantly. The third component is ProtoPie Connect, which serves as a bridge for communication between prototypes and external hardware, multiple screens, and APIs. Through Connect, designers can easily achieve real-time data exchange and interaction between prototypes and external devices. Finally, ProtoPie Cloud, this cloud platform is mainly used for sharing prototypes and team collaboration, allowing designers to share their design achievements and collaborate efficiently with team members. These four components together form the complete ecosystem of ProtoPie, providing a solid foundation for seamless integration of design and engineering.

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Image source: Speaker’s materials

ProtoPie Studio comes with a variety of interactive elements, providing great convenience for designers, allowing them to design various types of high-fidelity interactive and animated prototypes without writing any code. From basic interactive operations, such as clicking, double-clicking, rotating, zooming, color changes, etc., to more complex multimodal interactions, such as voice control and sensor responses, ProtoPie Studio provides comprehensive support.

Notably, the Send/Receive functionality in ProtoPie Studio has been widely used in cross-screen interaction and software-hardware interaction. Through this pair of elements, designers can easily achieve signal transmission between screens or between screens and external hardware, thus creating a rich variety of interactive experiences without writing any code.

In team collaboration scenarios, ProtoPie allows the creation of reusable interactive component libraries. Designers can build component libraries containing interactive functions in ProtoPie, such as vehicle control buttons and space bars, which are pre-designed in the library for quick reuse by designers across different projects while maintaining the original interactive characteristics.

Moreover, thanks to the ProtoPie Cloud platform, sharing prototypes has become exceptionally convenient. Designers only need to share the link of the ProtoPie prototype with the development team or colleagues, and the recipients can directly open and experience the interactive, dynamic prototype in their browser. This not only enhances collaboration efficiency but also ensures design consistency and real-time updates.

Furthermore, ProtoPie Cloud also supports recording the designed prototype as a delivery page. This interactive page not only fully showcases the entire high-fidelity interaction process but also details all parameters of the prototype, including position, distance, size, color, and animation curve parameters. This delivery method greatly simplifies the development team’s understanding of the design, reducing unnecessary communication and explanations, ensuring seamless integration between design and development.

The Necessity of Pre-validation

Previously, an OEM innovatively used varying sizes of bubble forms to present functional cards when designing HMI software, which seemed novel and aesthetically pleasing at the time. However, after the product was launched, user feedback fell far short of expectations. Through backend data analysis, they found that user click rates were significantly lower than anticipated, and they received a large number of negative feedback, with users generally finding the design difficult to use. The root of this problem was that they failed to conduct early validation of the design solution. After realizing the issue, although they quickly made updates through OTA, it was too late.

Automakers invest a significant amount of resources and time in vehicle model development, yet in the current highly competitive market environment, the margin for error is extremely low. If sufficient pre-validation is not conducted early on, the costs incurred when issues arise later will be substantial.

Before adopting ProtoPie, many automotive companies’ HMI development processes often followed this pattern: first, the product manager was responsible for preparing product documentation and creating low-fidelity wireframes to clarify the product framework. Subsequently, UI designers would independently design the UI based on these frameworks. Meanwhile, motion designers and 3D engineers were responsible for developing and producing animations. After completing these steps, the design solution would typically be integrated into a PPT or document for design review meetings, aimed at assessing usability and user experience.

After review and approval, the team would further generate detailed design specification documents and deliver them to the software development team for implementation. Only after the software development was completed would the solution be deployed to test benches, hardware systems, or actual vehicles to verify its actual usage effects and potential issues. However, if problems are discovered during this verification phase, going back to the design phase for modifications not only significantly delays project progress but also leads to a tremendous waste of resources.

The challenges in the previous process mainly included: first, PPTs are non-interactive and cannot vividly showcase design solutions, leading to cognitive discrepancies between designers and decision-makers; second, it was difficult to utilize the latest hardware and services (such as AI) for prototype demonstration and validation during the design process; third, the verification phase on test benches or actual vehicles was delayed, making it costly and time-consuming to iterate when issues were discovered, which could not meet the current market competition pace; fourth, developers found it challenging to accurately replicate interaction details, requiring designers to manually write extensive documentation and engage in repeated communication.

After using ProtoPie, the design process has been significantly optimized. First, high-fidelity layers and materials can still be produced in professional design software like Figma and Sketch; ProtoPie does not replace these tools but complements them. Subsequently, using ProtoPie’s plugin functionality, static high-fidelity layers can be directly imported into ProtoPie. On the ProtoPie platform, designers can easily add interactions and animations, and can insert particle effects (such as those created with AE) or 3D models (such as those created with Unity or other 3D engines) as needed. All these elements can be seamlessly integrated into ProtoPie, forming a complete, cross-screen, interactive HMI design solution. Through ProtoPie Connect, designers can easily connect external hardware, test benches, and AI technologies, completing the entire design display and validation during the design phase.

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Image source: Speaker’s materials

When the design team identifies issues, they can immediately make modifications in ProtoPie Studio, a process that typically takes only a few minutes, significantly improving efficiency compared to the previous need to wait for developer intervention. After modifications are completed, designers can generate delivery documents in ProtoPie with one click, which are clear and straightforward, directly provided to the development team, ensuring that the developed product is more mature and reliable.

After completing prototype design using ProtoPie Studio, designers can upload it to the ProtoPie Connect platform. Through ProtoPie Connect, prototypes can be easily pushed to various screen devices, including but not limited to Android car systems and any other screens that support browsers or Android systems. This wide compatibility makes prototype display more flexible and diverse.

Additionally, ProtoPie supports connections with hardware devices. In addition to hardware supported by built-in plugins, such as Arduino and G29, designers can connect via USB. At the same time, many hardware manufacturers have established partnerships with ProtoPie, providing hardware test benches compatible with ProtoPie, further expanding the testing and application scenarios of prototypes.

High-Fidelity Prototype Application Cases

The zero-code cross-screen interaction functionality enables smooth signal transmission through ProtoPie Connect, allowing ProtoPie Player to quickly and accurately display high-fidelity prototypes on various mobile devices, such as iPads, phones, and Android car systems. This process requires no coding, greatly enhancing design efficiency. The prototypes displayed on the screen below were all independently completed by ProtoPie, without any coding involved. Designers can achieve such excellent results without programming skills.

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Image source: Speaker’s materials

We can connect ProtoPie prototypes with test benches, a feature widely used by user experience research teams, innovation teams, and design teams. Through this connection, we can read various data in the driving simulation environment in real-time and feed it back to the ProtoPie prototype. In this scenario, interface elements such as the central control instrument are presented by the ProtoPie prototype, while all signals are connected to the prototype through ProtoPie Connect. Additionally, various commonly used driving simulation environments can be easily integrated, allowing us to quickly switch and compare different design solutions.

ProtoPie also supports connections with actual vehicles. In addition to being able to read OBD data from actual vehicles and display it in the prototype, we can also reverse control in-vehicle hardware by connecting to the vehicle domain controller. Of course, this feature is a more advanced application that requires designers to master ProtoPie proficiently to achieve such efficient interaction and control effects.

In terms of interacting with external APIs, ProtoPie can integrate third-party services such as facial recognition and head tracking APIs, enabling innovative functionalities through devices like webcams. These services can transmit data such as warning signals to the ProtoPie platform in real-time, ensuring interconnectivity between all signals.

Furthermore, as a powerful 2D rendering platform, ProtoPie also has the capability to integrate 3D content and web elements. Designers can import 3D engineering projects into ProtoPie, allowing them to focus solely on animation production, perspective switching, and rendering in the 3D engine. All interaction logic, vehicle control, and other components can be implemented through ProtoPie without writing any code. This feature enables designers to easily establish interactive relationships between 2D prototypes and 3D models, achieving innovative applications where 2D prototypes control 3D models.

(The above content is derived from the theme speech “Integrating AI and Smart Hardware into HMI Interaction Design” delivered by Li Li, Business Development Manager for the Asia-Pacific region at Studio XID Korea, Inc. on March 19, 2025, at the 5th China Automotive Human-Machine Interaction and Experience Design Innovation Conference.)

-END-

If you like this article, please give it a thumbs up

Integrating AI and Smart Hardware into HMI Interaction Design with ProtoPie

Yay

Leave a Comment