The design types involved in the IoT field cover software design, hardware design, and service design. As the design team of Alibaba’s IoT division, how can we establish a user experience monitoring system that meets business needs? How can we improve the efficiency of user experience collection? How can we effectively close the loop between user experience and business value? In this article, we will share our experiences in user research: how to build an “Experience Sensor Network” for IoT products.
User Experience Monitoring for IoT Products,
What Are the Challenges?
Traditional IoT development requires three steps: hardware development, software development, and cloud deployment. This requires teams with at least three technology stacks and a development process that lasts over three months. The long chain and multi-role business characteristics lead to several challenges in experience monitoring:
-
Many Offline Links: Most products in the IoT division have “broken links” in their usage chain, making it impossible to fully complete online tracking;
-
High Learning Curve: Users typically require a long learning path, as the products are highly specialized;
-
New Industry Models: The overall industry is in its early stages, and products may not have a fixed user path.
In the past, most experience monitoring systems were built on the premise of fully online product processes, single-role short links, and stable usage paths. User research primarily focused on models like Google’s HEART model, while the technical aspect was based on frontend data collection and server-side data construction. However, considering the business characteristics of IoT, we need to expand the types of feedback collected and broaden the channels for user data collection.
Building an “Experience Sensor Network”
Faced with the 100+ product matrix of the IoT division, how do we layout our experience monitoring system?
User research analysis often employs quantitative data collection, but different types of IoT products make quantitative data collection methods not particularly applicable. We classified and analyzed the data collection issues based on the operational environment of IoT technology products:
-
Cloud Loop: Using an Alibaba Cloud account in a browser can complete the entire usage chain, typically for workspace-type products. User operations are all performed in the cloud, and data can be collected through data collection methods. However, workspace products often involve heavy and fragmented operations, not typical single-link console products;
-
Cloud + Local: After completing configuration in the browser, secondary development is performed in the local environment. The cloud portion of these products is primarily based on console configuration. Relying solely on data collection can only provide insights into task completion rates and the final operational calls;
-
Cloud + Device: After completing configuration in the browser, the cloud directly sends commands to the IoT device. This is also a characteristic usage path for IoT products. Data collection can also provide insights into task completion rates from the cloud and device call situations, but the intermediate long-link usage process becomes a “black box”;
-
Cloud + Local + Device: After completing configuration in the browser, the SDK is downloaded to the local environment for secondary development, and finally burned into the IoT device. This is the most typical usage path for IoT products. Data collection can only provide insights from the beginning and end—i.e., task completion rates from the cloud and device call situations, making it difficult to locate issues in the intermediate “black box”.
Combining the aforementioned business characteristics, we have tried various research methods over the years, including data collection, user visits, focus groups, and surveys, and have established standard operating procedures for using more than 10 research tools. We compared the advantages and disadvantages of various tools and their investment returns as shown in the following figure.
Finally, we chose to use group tools like XSurvey/Uone for online operations that can be automated with minimal interference. For offline issues, we collaborated with the operations team to collect data through surveys/interviews/group robots/tickets. This formed an IoT-specific experience sensor network, continuously collecting experience data.
For example, with XSurvey, previously, the research on the experience of building tools could only be effectively obtained through offline workshops and user visits. We used XSurvey to shift the offline “manual” user research part to online “automated” research. We also set conditional triggers to ensure that only feedback from users who completed the “exploration” stage was collected, improving the quality of feedback. Compared to before, this not only reduced labor input by 90% but also increased the effective sample rate by 30%.
This experience data became the foundation for our monthly experience reports/special reports. The experience monthly report includes user feedback, usability experience data, core conversion indicators, etc., used to form a fixed monthly summary.
03
Tooling to Empower the Experience Value Loop
After completing the collection of experience data and the experience monthly report, to ensure that experience information can better impact the business, we formed a closed loop for experience information flow that includes reach-response-tracking after multiple rounds of collaboration.
In the operation of a mechanism, besides the rationality and effectiveness of its design, the maintenance cost of the mechanism is also very important. To reduce everyone’s burden, we adopted a tooling approach to ensure that the mechanism operates quickly and efficiently.
-
Reach: Using a low-code builder IoT Studio to generate experience monthly reports automatically. Key experience information is regularly pushed to groups via DingTalk robots;
-
Response: Empowering business stakeholders with specialized experience insight tools to tackle core issues;
-
Tracking: Discussing and collaborating with the Alibaba Cloud major customer experience team, using the same pool for tracking to avoid separate battles between experience design and customer service teams, achieving convergence.
The tooling approach reduced the maintenance cost of the experience mechanism by about 50%. It also doubled the reach rate of experience information and achieved cross-departmental reach and response. The implementation rate of experience needs reached 100%.
Taking the low-code IoT application building tool IoT Studio as an example, after the launch of the conditional questionnaire automatic collection tool, we collected user feedback and found that many users lacked the ability to develop mobile applications, yet their business required rapid mobile application development. Based on this feedback, we conducted in-depth follow-ups with core users, forming a complete research report that significantly accelerated the priority of launching the new version of the mobile workspace. After the workspace launch, it received a lot of user praise, with 60% of active users utilizing the new version of the mobile workspace, becoming one of the key features of IoT Studio.
03
Conclusion
There are various indicator systems and research tools in the industry, but they all aim to promote the realization of experience value in business. We should choose appropriate research tools based on our own business characteristics to complete the value loop of experience monitoring, ultimately bringing valuable user feedback to the business. Knowing is easy, doing is hard; let’s work together!
From May 14 to May 18, during the 2021 Alibaba Design Week, the first Usee Design Art Exhibition will be held! This 5-day feast of design and art will gather works from domestic and international art and design masters, numerous limited-edition collaborative products from artists, the debut of design black technology, and co-creation between Alibaba Design and products from various fields. The exciting live event cannot be missed!
