Table of Contents | 2024 Issue 2 Special Topic: Semantic Communication
Semantic Communication Based on Large Language Models: Current Status, Challenges, and Prospects
Image Semantic Communication System Based on Federated Learning
Semantic Communication System Integrating Knowledge Graphs
Semantic Communication Based on Internal Entity Information Enhancement
Knowledge Communication Aimed at Collaborative Evolution of Knowledge Bases
Research on Digital Twin Network Architecture Based on Semantic Cognitive Networks
Semantic Communication Framework for Smart Transportation Systems
Design and Implementation of a Multi-Task Semantic Communication Architecture
Information Theory and Coding for Function Computing
Semantic Coding Transmission Method for Human Action Videos
11【Semantic Communication】 Special Topic《Mobile Communications》 2024 Issue 2
Multi-Task Semantic Communication Coding Framework Based on Multiple Access
Lin Hang, Wu Yongpeng, Shi Yuxuan, Xu Mingkai, Zhang Wenjun
(School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240)
【Abstract】Semantic communication, as an emerging communication paradigm, has the potential to improve communication transmission efficiency. Based on this, a semantic communication framework MAMTSC is proposed under the context of semantic communication, combining multiple access and multi-task learning, which can perform multiple downstream tasks in a multiple access scenario. It uses a joint source-channel encoder and a joint source-channel-task decoder to multiplex semantic information between users and tasks, improving transmission efficiency. The framework designs a multiple access network capable of handling various downstream tasks and conducts simulation experiments in the context of two users, semantic segmentation, and depth estimation for image semantic communication. The effectiveness of the proposed framework is validated through performance comparison with rate-distortion metrics. Under low signal-to-noise ratio conditions, the proposed solution can achieve up to an 18% improvement in average intersection-over-union parameters for semantic segmentation compared to traditional coding methods.
【Keywords】Semantic Communication; Multiple Access; Multi-Task Learning
doi:10.3969/j.issn.1006-1010.20240108-0001
Classification Number: TN914.5 Document Code: A
Article Number: 1006-1010(2024)02-0070-07
Reference Format: Lin Hang, Wu Yongpeng, Shi Yuxuan, et al. Multi-Task Semantic Communication Coding Framework Based on Multiple Access[J]. Mobile Communications, 2024,48(2): 70-76.
LIN Hang, WU Yongpeng, SHI Yuxuan, et al. Multi-Task Semantic Communication Coding Framework Based on Multiple Access[J]. Mobile Communications, 2024,48(2): 70-76.
0 Introduction
The widespread adoption of mobile communication has continuously raised higher demands for communication transmission speed and quality. Therefore, the traditional communication development through external means such as stacking spectrum resources can no longer solve all problems[1]. This has also inspired relevant researchers to explore inwardly, initiating new thoughts from the design direction of communication systems. As a result, semantic communication has gained attention and research due to its high integration with artificial intelligence technologies[2] and its potential to enhance communication efficiency[3].
Semantic communication is an emerging communication paradigm that focuses on the semantic characteristics of transmitted information rather than precise bit reconstruction. At the same time, this paradigm is also referred to as a “task-oriented” communication method, which selectively extracts and compresses the semantic features of the original signal for downstream tasks to reduce the total amount of transmitted information and enhance resistance to channel noise[4]. Existing semantic communication works are primarily based on artificial intelligence frameworks, utilizing the superior performance of deep learning to achieve processing at the semantic feature level. Related works in artificial intelligence (AI) include studies on image semantic coding based on reinforcement learning[2], the role of end-to-end robust semantic communication systems supported by deep learning in combating semantic noise[5], etc. Model Division Multiple Access (MDMA)[6] technologies and Multi-access Block Attention Feedback (MBAF) coding[7] attempt to explore methods for improving information utilization among multiple users; semantic communication schemes based on extended rate distortion[8] and asynchronous training paradigms[9] utilize the same architecture to handle various downstream tasks and achieve certain performance improvements. It is worth noting that in the latest research on mobile communication system technologies, including the Internet of Things, the joint center receives data collected from multiple sensors, integrating the information obtained from all sensors to perform various tasks, including remote inference. Similar real-world applications involving multiple devices or multiple users and the concurrent requirements of multiple tasks are quite common in structurally similar mobile communication systems.
Based on this, this paper aims to combine semantic communication in multi-user scenarios with AI multi-task learning and attempts to propose a semantic communication framework that can execute multiple downstream tasks in a multiple access scenario. By setting up multi-user channel scenarios and multi-task requirements, it achieves the reuse of semantic information between users and tasks, thereby better completing related tasks under certain channel transmission rate requirements. The contributions of this paper are as follows: 1) A multi-access multi-task semantic communication framework (MAMTSC) is proposed; 2) A multiple access network capable of handling various downstream tasks is designed; 3) In examples of two users, semantic segmentation, and depth estimation for image semantic communication, the effectiveness is validated through performance comparison with rate-distortion metrics.
1 Related Work and Research Motivation
1.1 Multi-User Semantic Communication
For semantic communication systems in multi-user scenarios, many related works already exist. Similar to traditional communication using time division multiple access (TDMA) and frequency division multiple access (FDMA) technologies to multiplex system resources, Model Division Multiple Access (MDMA)[6] proposes using the model information space as a resource for semantic communication allocation and sharing model information to reduce bandwidth. This technology has been applied in video transmission[10] and point cloud data transmission[11]. MBAF codes[7] consider multi-access channels with feedback scenarios, attempting to establish collaborative working patterns among multiple encoders in a data-driven manner. Based on Type-Based Multiple Access (TBMA)[12-15] research, Information Bottleneck Type-Based Multiple Access (IB-TBMA)[16] uses the information bottleneck problem as a design criterion to jointly optimize the shared codebook and artificial neural network-based decoders. Unlike traditional communication, which can directly visualize the multiplexing of system resources, semantic communication attempts to find reusable resources among different users under certain constraints, such as the similarity of sources to a certain extent and the similarity of features required when user information is used to perform the same tasks, utilizing data-driven approaches.
1.2 Multi-Task Semantic Communication
Single-task semantic communication has achieved significant performance improvements through relevant research works. However, when downstream tasks change or when multiple models need to be stored to perform different tasks, the deep neural networks used in semantic communication need to be updated. To solve this problem, a unified framework is needed to serve different tasks. The semantic communication scheme based on extended rate distortion[8] derives a rate-distortion form and proposes a rate adjustment module to dynamically guide the trade-off between multiple AI tasks’ rates and distortions based on channel conditions. Multi-task learning itself has already developed to a certain extent in AI research, and thus, a comparative learning method has been proposed in combination with an asynchronous training paradigm[9] to achieve better performance in image classification and reconstruction tasks with fewer system training resources. The Unified Deep learning-enabled Semantic Communication (U-DeepSC) system[17] adjusts the number of transmitted symbols for different tasks based on the varying feature requirements of different tasks, significantly reducing model size while maintaining a certain level of performance.
1.3 Research Motivation
In multi-user semantic communication systems, the goal is to save communication system resources by reusing common semantic information among different users; in multi-task semantic communication systems, the goal is to merge models aimed at different tasks by reusing common semantic information required by different tasks. Both have made progress through the repeated use of semantic information in different areas of research. However, in practical scenarios, discussing only single-user multi-task or multi-user single-task semantic communication has certain limitations: for example, in IoT systems, multi-task requirements for single-device access or multi-device access for single-task requirements are specific application scenarios, while scenarios where multi-task requirements and simultaneous multi-device access coexist are more common in practice. Therefore, to achieve the reuse of semantic information among users and tasks, this paper designs the MAMTSC system framework.
2 System Model
3 Multi-Access Multi-Task Semantic Communication Scheme
3.1 Network Architecture
3.2 Loss Function Design
4 Simulation Experiments and Results
4.1 Experimental Setup
4.2 Experimental Results and Analysis
Figure 4 shows the performance of mIOU and rmse with the change of SNR under fixed CBR=0.142.
For the semantic segmentation task, it can be seen that the proposed scheme performs worse than traditional coding methods at high SNR, but the difference is not significant. As SNR decreases, the performance of traditional coding methods drops sharply, while the proposed scheme maintains relatively stable performance. Compared to DJSCC, it is found that the proposed scheme overall outperforms DJSCC, and both show relatively stable performance with changes in SNR.
For the monocular depth estimation task, the proposed scheme performs similarly to traditional coding methods at high SNR and outperforms them at low SNR. Compared to DJSCC, it is found that the proposed scheme performs better than DJSCC at high SNR but worse at lower SNR. The reason is that the network design of DJSCC is primarily aimed at image reconstruction, while the design of the proposed task decoder structure focuses on semantic segmentation, making the tasks of image reconstruction and depth estimation more closely related, thus DJSCC performs better at low SNR.
Additionally, for the depth estimation performance curve of the proposed scheme, there are abnormal performance drops at SNR=8, 9, and 10. This is because the training objective of the scheme overall considers both semantic segmentation and depth estimation tasks, leading to a situation where the performance improvement of the semantic segmentation task is smaller than the depth estimation performance drop, which is acceptable.
Figure 5 is a visual display of semantic segmentation results. The standard results are visualized based on the relevant files provided by the dataset. Compared to traditional coding methods, the semantic segmentation results of the proposed scheme show no significant difference at high SNR, while at low SNR, they significantly outperform traditional coding methods. Compared to DJSCC, the proposed scheme’s semantic segmentation results are overall better, and both results are relatively stable with changes in SNR.
Figure 6 is a visual display of depth estimation results. The standard results are visualized based on the relevant files provided by the dataset. The proposed scheme’s results are less accurate than traditional coding results at high SNR, but the overall image error is smaller; at low SNR, the estimation results significantly outperform traditional coding results. Compared to DJSCC, the proposed scheme performs better at higher SNR and worse at lower SNR, with DJSCC’s estimation results being more stable with changes in SNR.
4.3 Further Enhancements of the Scheme
The proposed multi-access multi-task semantic communication scheme performs well in semantic segmentation but does not meet expectations in depth estimation. The reason is that the structure design of the task decoder is not conducive to executing the depth estimation task. Therefore, if a more reasonable task decoder structure design can be found that better targets the required tasks, it can further enhance the scheme’s overall performance in multi-task execution.
It is observed that the depth estimation performance of the proposed scheme shows some abnormalities under high SNR, which prompts us to consider that the impact of different task performances on the overall system performance varies. Finding more suitable weighting coefficients to better balance performance variations among multiple tasks can improve the generalization ability of model scheme designs and expand application scenarios.
5 Conclusion
Faced with future communication applications requiring higher standards, this paper addresses the problem of reusing semantic information in multi-user and multi-task scenarios, proposing the idea of integrating both into the same theoretical framework, and constructing a system model for multi-access multi-task semantic communication, providing a possible model structure design scheme, and validating the effectiveness of the proposed scheme through simulation experiments. Compared to traditional coding methods, the proposed scheme can achieve up to an 18% improvement in mIoU parameters for semantic segmentation and a maximum improvement of 0.112 in rmse parameters for depth estimation under low SNR conditions.
It is also found in the experiments that the model and scheme design have potential directions for future improvement: the proposed scheme is suitable for two users and two tasks, and the application of related models in scenarios with greater user diversity and more diverse downstream tasks is worth studying; the design of encoders and decoders for different channels and downstream tasks may yield more generalized solutions in the future.
References:（Swipe up to browse）
[1] J Dai, P Zhang, K Niu, et al. Communication Beyond Transmitting Bits: Semantics-Guided Source and Channel Coding[J]. IEEE Wireless Communications, 2021, 30: 170-177.
[2] D Huang, F Gao, X Tao, et al. Toward Semantic Communications: Deep Learning-Based Image Semantic Coding[J]. IEEE Journal on Selected Areas in Communications, 2022, 41: 55-71.
[3] P Jiang, C Wen, S Jin, et al. Wireless Semantic Communications for Video Conferencing[J]. IEEE Journal on Selected Areas in Communications, 2022, 41: 230-244.
[4] Z Qin, X Tao, J Lu, et al. Semantic Communications: Principles and Challenges[EB/OL]. (2022-6-27)[2024-1-8]. https://arxiv.org/pdf/2201.01389.pdf.
[5] Q Hu, G Zhang, Z Qin, et al. Robust Semantic Communications Against Semantic Noise[C]. 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), 2022: 1-6.
[6] P Zhang, X Xu, C Dong, et al. Model division multiple access for semantic communications[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24: 801 – 812.
[7] E Ozfatura, C Bian, D Gündüz. Do not Interfere but Cooperate: A Fully Learnable Code Design for Multi-Access Channels with Feedback[C]. 2023 12th International Symposium on Topics in Coding (ISTC), 2023: 1-5.
[8] Z Sun, F Liu, Y Yang, et al. Multi-Task Semantic Communications: An Extended Rate-Distortion Theory Based Scheme[C]. 2023 IEEE International Conference on Communications Workshops (ICC Workshops), 2023: 1380-1385.
[9] Z Tian, HK Vo, C Zhang, et al. An Asynchronous Multi-Task Semantic Communication Method[J]. IEEE Network, 2023: Early Access.
[10] Z Bao, H Liang, C Dong, et al. MDVSC – Wireless Model Division Video Semantic Communication[EB/OL]. (2023-5-25)[2024-1-8]. https://arxiv.org/pdf/2305.15799.pdf.
[11] X Liu, H Liang, Z Bao, et al. Semantic Communications System with Model Division Multiple Access and Controllable Coding Rate for Point Cloud[EB/OL]. (2023-7-12)[2024-1-8]. https://arxiv.org/pdf/2307.06027.pdf.
[12] K Liu, AM Sayeed. Asymptotically optimal decentralized type-based detection in wireless sensor networks[C]. 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2004, 3: iii-873.
[13] G Mergen, L Tong. Type based estimation over multiaccess channels[J]. IEEE Transactions on Signal Processing, 2006, 56: 613-626.
[14] J Dommel, Z Utkovski, S Stańczak, et al. Joint Source-Channel Coding and Bayesian Message Passing Detection for Grant-Free Radio Access in IoT[C]. ICASSP 2020 – 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019: 8574-8578.
[15] J Dommel, Z Utkovski, O Simeone, et al. Joint Source-Channel Coding for Semantics-Aware Grant-Free Radio Access in IoT Fog Networks[J]. IEEE Signal Processing Letters, 2021, 28: 728-732.
[16] M Zhu, C Feng, C Guo, et al. Information Bottleneck-Inspired Type Based Multiple Access for Remote Estimation in IoT Systems[J]. IEEE Signal Processing Letters, 2022, 30: 403-407.
[17] G Zhang, Q Hu, Z Qin, et al. A Unified Multi-Task Semantic Communication System with Domain Adaptation[C]. GLOBECOM 2022 – 2022 IEEE Global Communications Conference, 2022: 3971-3976.
[18] M Yang, H Kim. Deep Joint Source-Channel Coding for Wireless Image Transmission with Adaptive Rate Control[C]. ICASSP 2022 – 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021: 5193-5197.
[19] L Chen, G Papandreou, F Schroff, et al. Rethinking Atrous Convolution for Semantic Image Segmentation[EB/OL]. (2017-12-5)[2024-1-8]. https://arxiv.org/pdf/1706.05587.pdf.
[20] N Silberman, D Hoiem, P Kohli, et al. Indoor Segmentation and Support Inference from RGBD Images[C]. European Conference on Computer Vision, 2012.
[21] E Bourtsoulatze D B Kurka, D Gündüz. Deep Joint Source-Channel Coding for Wireless Image Transmission[J]. IEEE Transactions on Cognitive Communications and Networking, 2018,5: 567-579. ★
Scan the QR code to readthe full paper on the web
★The original article was published in 《Mobile Communications》2024 Issue 2★
doi:10.3969/j.issn.1006-1010.20240108-0001
Classification Number: TN914.5 Document Code: A
Article Number: 1006-1010(2024)02-0070-07
Reference Format: Lin Hang, Wu Yongpeng, Shi Yuxuan, et al. Multi-Task Semantic Communication Coding Framework Based on Multiple Access[J]. Mobile Communications, 2024,48(2): 70-76.
LIN Hang, WU Yongpeng, SHI Yuxuan, et al. Multi-Task Semantic Communication Coding Framework Based on Multiple Access[J]. Mobile Communications, 2024,48(2): 70-76.
Author IntroductionLin Hang（orcid.org/0009-0006-7826-9466）：Undergraduate student at the Department of Electronic Engineering, Shanghai Jiao Tong University, recommended for direct admission to the master’s program in Information Engineering, with a primary research focus on semantic communication.Wu Yongpeng（orcid.org/0000-0003-1716-1882）：Professor at Shanghai Jiao Tong University, IEEE Senior Member, Chair and committee member of international conferences such as IEEE ICC, Globecom, VTC; research focuses on space-time wireless communication theory and key technologies; has led multiple research projects including National Natural Science Foundation, National Key R&D Sub-Projects, and projects from ZTE Corporation and State Grid; recipient of various honors including Outstanding Youth Fund from National Natural Science Foundation, IEEE Communications Society Asia-Pacific Outstanding Young Researcher Award, Ji Hanbing Young Teacher Award, and “Youth Lifting Talents Program” from China Association for Science and Technology.Shi Yuxuan:PhD student at the School of Cyberspace Security, Shanghai Jiao Tong University, main research focus on information theory and coding, semantic communication.Xu Mingkai:PhD student at the Department of Electronic Engineering, Shanghai Jiao Tong University, main research focus on semantic communication.Zhang Wenjun:Professor at Shanghai Jiao Tong University, Chief Scientist at the National Engineering Research Center for Digital Television (NERC-DTV), Director of the Future Media Network Collaborative Innovation Center (CMIC); main research focuses on video coding and wireless transmission, multimedia semantic analysis and broadcasting/broadband network integration; major contributor to the China DTTB standard (DTMB).
《Mobile Communications》 submission method is online submission
Please log in to the web submission system
Link address:http://ydtx.cbpt.cnki.net
Highlights【2024 Special Call for Papers–Issue 6】6G Wireless Technology【2024 Special Call for Papers–Issue 7】6G Endogenous Intelligent Technology【2024 Special Call for Papers–Issue 8】Integration of 6G and AITable of Contents | 2024 Issue 3 Current Topic: 6G Sensing and Computing IntegrationTable of Contents | 2024 Issue 2 Current Topic: Semantic CommunicationTable of Contents | 2024 Issue 1 Current Topic: Technology for 6G Ground-Space Integration NetworksCollection | 2023 Issues 1-12 Special Paper Collection
#Scan to follow us#《Mobile Communications》Interpreting Communication through Papers
《Mobile Communications》 magazine is supervised by China Electronics Technology Group Corporation, hosted by the Seventh Research Institute of China Electronics Technology Group Corporation. It is a dual-effect journal in the Chinese journal matrix, a fine electronic journal by the Ministry of Industry and Information Technology, a source journal for China Science and Technology Papers Statistics, included in the high-quality scientific journal grading directory of the Information and Communication field of the Chinese Institute of Communications, and a high-quality scientific journal grading directory of the Chinese Institute of Electronics. Domestic continuous publication number: CN44-1301/TN, International Standard Serial Number: ISSN1006-1010, Postal delivery code: 46-181.