Article Title: A Method for Encrypted Video Recognition Based on HTTP/2 Multiplexing Features
All Authors: Wu Hua, Luo Hao, Zhao Shishun, Liu Songtao, Cheng Guang, Hu Xiaoyan
First Affiliation: Southeast University, School of Cyberspace Security
Publication Date: 2025, 36(7): 3375–3404
Abstract
The rise of video application platforms has enabled rapid dissemination of videos, permeating various aspects of social life. However, the videos transmitted over the internet also include harmful content, necessitating accurate identification of encrypted harmful videos for effective cyberspace security regulation. Existing methods collect traffic data at major network access points, extract features of encrypted video traffic, and identify transmitted harmful videos based on a library of harmful videos through traffic feature matching. However, with the update of video encryption transmission protocols, the newly deployed HTTP/2 protocol, which utilizes multiplexing technology, has rendered traditional traffic analysis methods based on HTTP/1.1 transmission features ineffective for identifying encrypted videos transmitted via HTTP/2. Furthermore, current research primarily focuses on videos with fixed playback resolutions, rarely considering the impact of adaptive resolution switching during streaming playback on recognition. To address these issues, this paper analyzes the principle of audio and video data length offset when video platforms transmit videos using the HTTP/2 protocol and proposes a method to accurately correct and restore multiplexed encrypted data to the combined audio and video data unit lengths, thereby constructing an accurately restored encrypted video correction fingerprint. Subsequently, utilizing the encrypted video correction fingerprint and a large plaintext video fingerprint library, a sliding matching mechanism for video correction fingerprints and an encrypted video recognition model based on Hidden Markov Models and the Viterbi algorithm are proposed. This model employs dynamic programming to resolve issues arising from adaptive resolution switching, achieving recognition accuracy rates of 98.41% and 97.91% for encrypted videos with fixed and adaptive resolutions, respectively, in a real fingerprint library scenario of 400,000 instances from Facebook and Instagram. The method’s universality and generalization were validated using three video platforms: Triller, Twitter, and Mango TV. Comparisons with similar works in terms of recognition effectiveness, generalization, and time overhead further validate the high application value of the proposed method.
Scan to read the full text
Video
/Content/SelectionCONTENTCONTENT1 Method Overview
This paper employs a plaintext fingerprint recognition method for encrypted videos as illustrated in Figure 1. First, an automated collection system is utilized to gather plaintext video fingerprints and encrypted video transmission data in real playback scenarios, constructing a plaintext video fingerprint library. Next, data features are extracted from the encrypted video data stream, and after processing through the correction restoration model, an encrypted video correction fingerprint is constructed. Finally, based on the encrypted video correction fingerprint and the plaintext video fingerprint library, a suitable video recognition method is designed to identify the content of encrypted videos.
Figure 1 Overall Architecture of the Plaintext Fingerprint Recognition Method for Encrypted VideosIn the recognition process shown in Figure 1, the plaintext fingerprints obtained during the first step of constructing the plaintext fingerprint library are accurate. However, the second step of restoring the correction fingerprint from the ciphertext must consider various influencing factors during the encryption and transmission of audio and video data. Different influencing factors and correction restoration techniques yield different restoration results, leading to uncertainty in the corrected fingerprint results. This uncertainty further affects the accuracy of video recognition in the third step. Therefore, it is necessary to analyze the impact of various network protocols, especially the HTTP/2 protocol, on the corrected restoration fingerprint.The key technical challenges that this method needs to address are as follows:
(1) The encapsulation and segmentation of audio and video segments at various protocol layers lead to a random increase in the length of encrypted data compared to the original plaintext length.
(2) The changes in stream characteristics due to the HTTP/2 protocol necessitate a new method for constructing video correction fingerprints.
(3) Adaptive resolution switching and user-driven playback cause video segments to be transmitted out of order.① Automated Construction Method for Large Video DatasetsTo address the lack of real large video datasets in the field of video recognition and to validate the method in real scenarios, this paper designs and implements an automated construction system for large video datasets, as shown in Figure 2. This system includes fully automated collection of plaintext and encrypted video transmission data, enabling the automatic construction of large real video datasets.
Figure 2 Automated Construction System for Large Video DatasetsPlaintext Video Fingerprint Collection: The plaintext video fingerprint refers to the length sequence of plaintext video data segments. The plaintext fingerprint information of the video comes from the video description file. As shown in Figure 2, the plaintext fingerprint automatic collection system mainly includes a video URL collection module, MPD file download module, video index segment download module, and plaintext fingerprint parsing module. First, the URL collection module gathers video URLs of different categories, durations, and resolutions from the target platform, deduplicates them, and sends them as input to the MPD file download module. The MPD file is downloaded using the video URL and related signature information, and after parsing the MPD file, key information is extracted. Finally, based on the extracted information, the index segments are unpacked to extract the plaintext video fingerprints, and the key information of the video extracted during this process is stored as a whole in the plaintext video fingerprint library.Encrypted Video Transmission Data Collection: The encrypted video transmission dataset is collected in a real network playback environment and is used as the training and testing set in this study. To comprehensively validate the effectiveness of the recognition method, the system collects encrypted transmission data from two terminal playback scenarios: personal computer (PC) web clients and mobile clients. Using the aforementioned automated collection system, continuous and uninterrupted automated collection of encrypted video transmission data from different video terminals can be achieved. This paper successfully constructed a large dataset containing 400,000 plaintext video fingerprints and encrypted video transmission data. The relevant information of this dataset is shown in Table 1.Table 1 Statistics of Relevant Information for the Large Video Dataset
② Method for Constructing Encrypted Video Correction Fingerprints Based on HTTP/2 Protocol
The overall framework of the proposed method for constructing encrypted video correction fingerprints is shown in Figure 3. This method is mainly divided into two stages.
-
The first stage is the model training stage, where encrypted transmission data and corresponding labels from the plaintext fingerprint library are used to train the TLS correction model and the HTTP/2 correction model sequentially, based on the principles of data encryption encapsulation and transmission by TLS and HTTP/2, for correcting encrypted transmission data.
-
The second stage involves using the two trained correction models from the first stage to accurately correct and restore the encrypted video transmission data, thereby constructing the correction fingerprint for the encrypted video.
Figure 3 Encrypted Video Correction Fingerprint Construction Framework
In this paper, these continuously transmitted combined response data are collectively referred to as Combined Data Unit (CDU). Due to the use of multiplexing technology, the audio and video packets in these CDUs are mixed and transmitted after encryption, making it impossible to distinguish them. Therefore, this paper treats the CDU as a whole for length correction and restoration.
Figure 4 illustrates the encapsulation process of audio and video data CDUs. When entering network link transmission, audio and video data CDUs are sequentially processed by protocols such as HTTP/2, TLS, and TCP, undergoing corresponding segmentation, compression, encryption, and padding operations, and adding corresponding protocol headers and control information to ensure stable and reliable transmission of CDUs. To achieve the goal of correcting and restoring encrypted data, it is necessary to subtract the length of all additional control information added from the actual transmitted payload length.
Figure 4 Encapsulation Process of Video Data Packets CDU
Audio and video combined data units CDU may contain one or more combined audio and video data segments. Due to the multiplexing feature of HTTP/2, multiple data streams can be transmitted simultaneously over the same TCP connection (as shown in Figure 5), making it necessary to accurately delineate the audio and video packet data CDUs of the encrypted data stream for subsequent correction and restoration work, ensuring that each delineated group of encrypted data contains only one CDU.
Figure 5 Precise Division of Encrypted Audio and Video Packet Data Units CDU
To improve the accuracy of model predictions, this paper further processes the feature vectors used for training the prediction model. A continuous, equal-length TLS record that appears more than a threshold in a stream is defined as a frequent item of that stream, and then all TLS record sequences corresponding to CDUs in the training set undergo a frequent item merging operation. This merging operation allows the feature values contained in the feature vector to be closer to the actual DATA frame lengths and significantly reduces the length of the feature vector. As shown in Figure 6, this method can significantly enhance the model’s accuracy.
Figure 6 Comparison of Prediction Effects of Various Machine Learning Models
③ Encrypted Video Recognition ModelWhen designing the matching recognition algorithm for the correction fingerprint sequence and the plaintext fingerprint sequence, it must be applicable to actual situations. Figure 7 illustrates the encrypted video matching recognition algorithm proposed in this paper.
Figure 7 Principle of the Encrypted Video Matching Recognition Algorithm
As shown in Figure 7, the encrypted video recognition algorithm is divided into two stages: the sliding matching stage of the video correction fingerprint and the video recognition stage.
- In the sliding matching stage of the video correction fingerprint, based on the sliding window concept, the encrypted video correction fingerprints are stored in the plaintext video fingerprint library for matching, outputting possible matching results for each correction fingerprint fragment.
- In the video recognition stage, using the obtained matching results and transition probability matrices, a Hidden Markov Model is constructed, and finally, the Viterbi algorithm is employed to solve the model, calculating the plaintext video fingerprint sequence most likely to match the encrypted video correction fingerprint sequence, thereby achieving the recognition of encrypted videos.
Figure 8 illustrates the principle of obtaining candidate fingerprint fragment combinations through sliding matching of a correction fingerprint fragment on the plaintext fingerprint. To achieve one-to-many matching between correction fingerprint fragments and plaintext fingerprint fragments, this paper adopts a method of dynamically adjusting the sliding window size for matching. First, the sliding window size WS is set to 1–n, and it slides over the plaintext fingerprints in the video fingerprint library, where n is related to the content distribution mechanism of the video platform and needs to be obtained through actual measurement. During the sliding process, the fingerprint fragments within the sliding window P form a fingerprint fragment combination that matches the correction fingerprint fragment within an error range. Subsequently, the correction fingerprint fragment is matched with all fingerprint fragment combinations within the error range. The matching error range [−T, T] needs to be set based on the correction error of the correction fingerprint fragment. Finally, this paper derives the matching probability for each matched fingerprint fragment combination based on the matching error.
Figure 8 Principle of Sliding Matching Between Correction Fingerprint Fragments and Video Plaintext Fingerprints
This paper generates a key-value pair database in memory based on the principle of sliding combinations of video plaintext fingerprints illustrated in Figure 8. In the key-value pair database, each key corresponds to the length of a plaintext fingerprint fragment combination, and the value corresponds to the descriptive information of the plaintext fingerprint fragment combination (e.g., video ID, resolution, combined segments, etc.). For key-value pairs with the same key, this paper employs a linked list structure for storage. The structure of the key-value pair database generated from video plaintext fingerprints is shown in Figure 9.
Figure 9 Matching Principle of Correction Fingerprints and Plaintext Fingerprints in Key-Value Pair Database
To further reduce fingerprint matching time and accelerate matching speed, this paper divides the entire matching process of correction fingerprints into full matching and fast matching two stages. Through these two stages of matching, the number of matched fingerprints can be significantly reduced, quickly obtaining matching results and probabilities for each correction fingerprint fragment. The specific process is illustrated in Figure 10.
- The purpose of the full matching stage is to use the first few fragments of the correction fingerprint for matching, thereby filtering out all possible candidate videos from the massive fingerprint library and forming a small candidate plaintext video fingerprint library.
- In the fast matching stage, only the remaining fragments of the correction fingerprint need to be matched within this small candidate plaintext video fingerprint library to obtain matching results and probabilities for all correction fingerprint fragments.
Figure 10 Phased Matching Process of Correction Fingerprints
This paper constructs a Hidden Markov Model based on observable state sets, hidden state sets, state transition matrices, emission probability matrices, and initial state matrices—known as a fence network (as shown in Figure 11), which can subsequently be solved using the Viterbi algorithm, the most commonly used method for solving Hidden Markov Model decoding problems, to derive the plaintext fingerprint sequence corresponding to the encrypted video correction fingerprint, thereby recognizing the encrypted video.
Figure 11 Recognizing Encrypted Videos Based on HMM2 Experimental Analysis
Since the scale of the fingerprint library significantly impacts the accuracy of video recognition, this paper conducts accuracy tests of the video recognition method using collected real encrypted video transmission data across different fingerprint library scales. Additionally, existing research has only validated recognition accuracy in fixed resolution scenarios, but in actual playback, video resolution may adaptively switch due to changes in network conditions, posing challenges for recognition. To validate the applicability of this method in real playback environments, tests were conducted in both fixed and dynamic resolution scenarios. Table 2 presents the accuracy results of encrypted video recognition in different fingerprint library scenarios under fixed resolution conditions. Table 3 presents the accuracy results of encrypted video recognition in different fingerprint library scenarios under dynamic resolution conditions. The method proposed in this study demonstrates excellent performance across various evaluation metrics in different fingerprint library recognition scenarios and different playback modes.
Table 2 Encrypted Video Recognition Results Under Fixed Resolution Scenarios
Table 3 Dynamic Resolution Scenario Encrypted Video Recognition Results
To validate the universality and generalization of this method across different video platforms and environments, this paper supplemented the selection of three domestic and international video platforms: Triller, Twitter, and Mango TV, based on Facebook and Instagram. Following the method flow introduced earlier, first, plaintext fingerprints of test videos from the three platforms were collected and stored in a large plaintext video fingerprint library. Next, for each platform, 50 videos were selected for a single playback data collection to train the parameters of the platform video recognition model. Subsequently, large-scale test data was collected. Finally, video recognition experiments were conducted on the test data from each platform. In the fixed resolution playback scenario, for Triller, Twitter, and Mango TV, experiments selected 3184, 3000, and 1200 videos, respectively, with durations ranging from 30 to 300 seconds, collecting single playback data as test data, with recognition results shown in Table 4.
Table 4 Recognition Results of New Platforms Under Fixed Resolution Scenarios
Given the diverse triggers for video resolution switching, this paper also uses network simulation tools NetEm and tc to control network parameters and simulate network fluctuations. To simulate various ways that trigger adaptive switching in real scenarios, different resolution switching environments were set for the three platforms. The recognition results obtained from the experiments are shown in Table 5.
Table 5 Recognition Results of New Platforms Under Dynamic Resolution Scenarios
It can be seen that using the method proposed in this paper, both in fixed resolution playback scenarios and dynamic resolution playback scenarios, the three platforms exhibited good recognition results.
To validate the matching efficiency of this method in different scales of plaintext video fingerprint libraries, experiments were conducted using 800 video playbacks for fingerprint libraries ranging from 0.5 to 400,000. The average time taken to recognize each video was recorded and calculated, with results shown in Figure 12. It can be observed that although the average recognition time for a single video increases with the scale of the fingerprint library, the overall time overhead of this method remains small across different fingerprint library scales. Even at a scale of 400,000 fingerprints, the recognition time for a single video is only 133.32 ms, indicating that this method can be applied in real large-scale fingerprint library matching scenarios.
Figure 12 Average Matching Recognition Time Overhead of This Method in Different Scale Fingerprint Library Recognition Scenarios3 Conclusion and OutlookTo address the challenges posed by the widespread application of the new HTTP/2 protocol for encrypted video recognition, this paper proposes a method for encrypted video recognition based on HTTP/2 traffic multiplexing features. This method consists of two main steps. The first step involves accurately correcting and restoring multiplexed HTTP/2 encrypted data to the lengths of audio and video data unit combinations, achieving a correction error of over 99.70% within a range of 0.5‰. This high-precision fingerprint correction and restoration method enables accurate video matching within large fingerprint libraries. The second step designs a resolution-adaptive encrypted video recognition method based on encrypted video correction fingerprints and plaintext video fingerprints. Experimental results indicate that this recognition model achieves high recognition performance in a real large fingerprint library of 400,000 instances, exceeding 97% accuracy, over 99% precision, over 97% recall, and over 99% F1 score in both fixed resolution and adaptive resolution scenarios. The research and experiments in this paper are based on a real large video fingerprint dataset, and comparisons with similar works also demonstrate the practicality of this method.The method proposed in this paper is based on the analysis of traffic data at key network access points and does not require cooperation from video platforms, making this technology independent of video platform constraints and easily deployable. The constructed plaintext video fingerprint library is oriented towards dynamic real networks, capable of collecting all resolutions of a video’s plaintext fingerprints within 1–2 seconds, requiring minimal network bandwidth and storage space, and can be expanded and updated at any time based on the identification of harmful videos. Therefore, this method has high application value. Furthermore, this method does not require decryption of network application layer video data; it performs fine-grained content recognition of harmful videos in the fingerprint library based solely on the traffic characteristics of video transmission. Thus, this method balances the fine-grained management of harmful videos while respecting the privacy of normal users who transmit data end-to-end through encryption. The convenience of practical deployment and consideration for normal user privacy make this method capable of meeting the internet information sharing needs under the premise of refined network management.This method utilizes the protocol characteristics during video data transmission, combining domain knowledge with machine learning to achieve precise recognition of HTTP/2 encrypted video data. Currently, the coexistence of HTTP/1.1, HTTP/2, QUIC, and HTTP/3 protocols on the internet is gradually evolving, with HTTP/2 and HTTP/3 expected to dominate in the future. In the face of complex network environments, enhancing the universality of methods while ensuring their efficiency and practicality, using domain knowledge-driven artificial intelligence for general encrypted video recognition will be a future research direction and hotspot in this field.Author IntroductionWu Hua, Ph.D., Associate Professor, Doctoral Supervisor, CCF Professional Member, main research areas include encrypted traffic analysis, cyberspace security, and network situational awareness.
Luo Hao, Master’s Student, main research areas include encrypted traffic analysis and encrypted video recognition.
Zhao Shishun, Master’s Student, main research areas include encrypted traffic analysis and encrypted video recognition.
Liu Songtao, Master’s Student, main research areas include machine learning and encrypted traffic classification.
Cheng Guang, Ph.D., Professor, Doctoral Supervisor, CCF Distinguished Member, main research areas include cyberspace security monitoring and protection, big data analysis of network traffic, botnets, and APT attack detection.
Hu Xiaoyan, Ph.D., Associate Professor, Doctoral Supervisor, CCF Professional Member, main research areas include encrypted traffic analysis, cyberspace security, and future network architecture.