Imagine a system that can accurately determine whether you are cooking, cleaning, or exercising, solely based on sensor data from your smartphone. The research team at Apple is making this scenario a reality by developing a novel multimodal sensor fusion method based on large language models (LLMs) for human activity recognition.This research was conducted in collaboration with researchers from MIT and Johns Hopkins University, exploring how to leverage the world knowledge of LLMs to fuse time-series sensor data for zero-shot and few-shot activity classification. The paper is currently published on the arXiv preprint platform.
Challenges of Multimodal FusionHuman activity recognition has always been a core challenge in fields such as human-computer interaction and health monitoring. The data generated from our daily activities typically comes from various sensors, such asaudio devices, accelerometers, and gyroscopes.Traditional methods require a large amount of aligned multimodal data for training to learn a shared embedding space. However, obtaining high-quality, aligned multimodal data is extremely difficult. For example, activity samples that contain rich audio and motion sensor data with accurate annotations are quite rare in real-world environments.The researchers pointed out in the paper: “Learning a descriptive joint embedding space for time-series signals requires a large sample size, and scaling to multimodal is very challenging due to the scarcity of aligned paired data.”Methods such as contrastive learning mayactively suppress learning related but orthogonal information, while generative methods are limited by the lack of rich descriptive annotations for sensor data.Research Method: How to Achieve Sensor Fusion with LLMs?The research team proposed an innovativelate fusion method that completely avoids the traditional training requirements for cross-modal alignment. Their core idea is to leverage the existing world knowledge and reasoning capabilities of LLMs to directly fuse the processed results from different sensors.The researchers selected 12 categories of daily activity data from the Ego4D dataset, including household activities (such as vacuuming, cooking, and laundry) and sports activities (such as basketball, soccer, and fitness). Each sample consists of a 20-second multimodal data segment.
Figure 1 Model architecture diagram created for prompting (Prompt).This diagram illustrates the entire system’s workflow. On the left is the raw multimodal time-series data (audio and IMU). The middle section consists of variousmodality-specific models that process their respective data and output predictions (such as audio descriptions, audio labels, and IMU activity labels). On the right is thelarge language model (LLM), which receives the outputs from all modality models at each time step and formats them into a structured text prompt, then performs reasoning to ultimately output high-level activity classification results.The data processing workflow consists of three steps:First, audio data is processed through MS CLAP and VGGish models to generate audio descriptions and labels. Motion data (from the IMU’s 3-axis accelerometer and 3-axis gyroscope) is processed through a specially trained activity classification model that can only output six basic actions (such as walking, running, standing, etc.).Second, the researchers also synthesized additional contextual information, such as indoor/outdoor environment and heart rate zones, to simulate sensor data that might be obtained in real applications.Finally, all modality prediction results are organized in chronological order into prompts and input into the LLM for reasoning. The researchers tested two LLMs:Gemini-2.5-pro and DeepSeek-R1-Distill-Qwen-32B.Zero-shot and Few-shot Learning: How Does LLM Fusion Perform in Practice?The research team designedclosed-set and open-set evaluation schemes. In the closed-set evaluation, the model needs to select the most likely one from the given 12 activity categories; in the open-set evaluation, the model can freely generate activity descriptions without being restricted by a preset list.
Table 1 High-level activity classification performance under single-sample closed-set setting (95% confidence interval). This table shows the performance of the two LLMs (Gemini-2.5-pro and Qwen-32B) under different modality combinations in the evaluation setting ofsingle-sample, closed-set (i.e., selecting from 12 fixed options). It compares the accuracy and macro F1 scores of various scenarios, including using all modalities (audio descriptions + audio labels + IMU predictions + additional context), missing additional context, using only audio descriptions, only audio labels, and only IMU predictions.
Table 2 High-level activity classification performance under single-sample open-set setting (95% confidence interval). This table is similar to Table 1 but evaluates theopen-set scenario, where the LLM does not need to select from a preset list but can freely generate activity descriptions. To quantify the results, the researchers mapped these freely generated answers back to the 12 preset categories using the Qwen-32B model for scoring. This table shows the performance of the LLM in a more free and challenging task.The results show that in the few-shot closed-set classification, Gemini-2.5-pro achieved68% accuracy and 66% macro F1 score when using all modality information (audio descriptions, audio labels, IMU activity predictions, and additional context), significantly higher than the random guess rate of 8.3%.Even in the more challenging open-set evaluation, the model’s performance greatly exceeded the random baseline, demonstrating the potential of LLM-based fusion in difficult tasks.Analysis of Modality Contributions shows that audio descriptions are the most useful for overall activity prediction. While IMU activity predictions have limited overall performance improvement (partly due to their limited output space), they play a crucial role in correct predictions in certain cases, especially in high-motion activities (such as soccer).The research also found that the addition of synthesized contextual information further improved performance, indicating that even simple sensor-derived information (such as heart rate ranges) can enhance activity recognition accuracy.Limitations and Future DirectionsThe research also revealed the limitations of the current method.Errors in modality-specific model predictions can significantly affect the reasoning results of the LLM. For example, one case in the paper showed that when the audio model incorrectly identified the breathing sound during exercise as an animal sound like “a horse walking,” the LLM was misled and ultimately predicted “exercising” as “playing with pets.” Similarly, when the IMU model failed to detect subtle movements, the LLM tended to predict static activities (such as “eating” or “reading”).Another limitation is that the IMU classification model used has a very limited output space, only able to predict six basic activities. The researchers noted that more robust and flexible motion models could help further distinguish between high-motion and low-motion activities.Additionally, the scale of the dataset evaluated in the current study is relatively small, containing only 12 activity categories. The model’s generalization ability still needs further validation in broader real-world application scenarios.This research opens new avenues for processing multimodal time-series data. Future work will expand the evaluation data and modality-specific models and explorestrategies for training LLMs to acquire targeted reasoning skills for better modality integration.Paper Information:Demirel, I., Thakkar, K., Elizalde, B., et al. “Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition.” arXiv preprint arXiv:2509.10729 (2025).
About Us
21dB Acoustic People is a technology media under the Suzhou Electroacoustic Industrialization Base of the Institute of Acoustics, Chinese Academy of Sciences, focusing on new acoustic technologies, audio testing and analysis, acoustic market research, acoustic entrepreneurship incubation, intellectual property services, etc.
Cooperation Promotion
Article Submission ｜ Entrepreneurship Support ｜ Intellectual Property Services ｜ Technology TransferPlease send your requirements to the following email, and we will send a dedicated person to contact you[email protected]
Copyright Statement
The content of this article is based on publicly available information from the journal paper, intended for academic exchange and information sharing, and does not bear responsibility for its copyright.All technical details, experimental data, and conclusions are based on the original paper. If there are issues of infringement, privacy, etc., please contact us for timely deletion. Contact WeChat:acoustic21dB
