Abstract: With the continuous development of the internet and the increasing demand for security, encryption technology has become the preferred choice for ensuring traffic security. However, it has also led to a surge in encrypted malicious traffic. In the face of a complex and ever-changing network environment, the ability to quickly identify malicious traffic without decryption is of great significance for enhancing network security capabilities. This article focuses on the classification of malicious traffic as the research basis, sorting through currently popular technologies for analyzing and identifying encrypted malicious traffic, with a focus on traffic identification methods based on unidimensional and multidimensional features. It discusses the application research of cutting-edge technologies in the field of encrypted malicious traffic analysis and points out directions for future research.
Encryption is an important means of protecting privacy, capable of safeguarding data from being intercepted and preventing attackers from stealing information, applications, or passwords. In recent years, traffic encryption has been regarded as an important trend in the development of the internet, especially during the global outbreak of the COVID-19 pandemic in 2020, which increased the frequency of remote work, online teaching, and remote meetings, thereby exacerbating the demand for traffic encryption. Generally, encryption is considered synonymous with security; however, this perspective is relative. In a complex internet environment, systems are susceptible to external attacks, and simple encryption methods cannot guarantee the security attributes of confidentiality, integrity, and availability of information. When facing traffic, attackers may use encrypted traffic to carry out malicious attacks, resulting in more destructive behaviors. According to the international research organization Gartner, by 2020, over 60% of enterprises would be unable to effectively decrypt HTTPS traffic, and the means to counter these threats would be constrained by anti-decryption systems, hiding over 70% of network malware within encrypted traffic. According to a Cybersecurity Ventures survey, the time between ransomware attacks on enterprises decreased from every 14 seconds in 2019 to every 11 seconds in 2021, making ransomware the fastest-growing type of cybercrime. In 2021, global losses due to ransomware were estimated to reach $20 billion, far exceeding the $325 million in 2015. Therefore, timely and rapid identification and analysis of encrypted malicious traffic is of great importance for enhancing network security resilience and purifying cyberspace.
1. Current Research Status
For encrypted traffic, the mainstream attack analysis methods include post-decryption analysis and non-decryption analysis. Due to strict legal regulations regarding privacy protection during the decryption process, the industry currently mainly uses non-decryption methods to analyze attack behaviors. Moreover, there have been some research achievements in detecting malicious traffic directly from encrypted traffic without decryption. Pan Wubin and others summarized the architecture of encrypted traffic identification, detailing the types of encrypted traffic identification, such as protocol, application, and service, and providing an overview of existing encrypted traffic identification technologies while analyzing and comparing them from multiple perspectives. Wang Ying and others established an encrypted traffic detection framework and analyzed encrypted traffic monitoring using key technologies and related methods. Luo Ziming and others introduced the characteristics of the Transport Layer Security (TLS) protocol and traffic identification methods, proposing a machine learning-based distributed automated system for detecting encrypted malicious traffic, utilizing multiple traffic features for in-depth analysis and comparing the performance of related algorithms through experiments. Zeng Yong and others reviewed various methods for identifying encrypted malicious traffic, including those based on machine learning and cryptography, providing significant guidance for identifying encrypted malicious traffic. The aforementioned experts and scholars have their own insights on the research of encrypted traffic. This article organizes the current status of encrypted traffic identification technologies, focusing on various identification methods for the features of encrypted malicious traffic, and discusses the application of cutting-edge technologies in the field of encrypted malicious traffic analysis, pointing out directions for future research work.
2. Classification of Malicious Traffic
Overall, encrypted traffic can be divided into encrypted normal traffic and encrypted abnormal traffic. In most cases, encrypted abnormal traffic can further be divided into benign encrypted abnormal traffic (such as traffic anomalies caused by changes in certain parameters or an increase in access) and malicious encrypted traffic. Among the classifications of encrypted traffic, encrypted malicious traffic is the most difficult and dangerous, hiding many known or unknown threats. By refining the differentiation of traffic, targeted control measures can be taken to effectively identify, analyze, and block encrypted malicious traffic, which is of great significance for enhancing network security protection capabilities. Malicious traffic can be summarized into the following three types based on attack behaviors.
(1) Malicious software using encrypted communication. This category mainly refers to malicious code and malware that use encrypted communication to disguise or hide plaintext traffic characteristics to evade detection by security products and human operators. For example, Trojans, infected viruses, worm viruses, and malicious downloaders may disguise or hide attack behaviors using encryption.
(2) Malicious attack behaviors on the encrypted channel side. This category mainly refers to attackers who initiate attacks using already established encrypted channels. Attack behaviors include scanning, probing, and brute-force cracking.
(3) Malicious or illegal encrypted applications. This category mainly refers to certain malicious or illegal applications that use encrypted communication. Compared to classifying malicious traffic based on attack behaviors, academia tends to focus more on classifying malicious traffic based on specific characteristics such as content features, data flow features, and network connection behavior features. Different characteristics have their own typical traits. Content features include unique values in the malicious traffic protocol segment and special character sequences contained in the payload, while data flow features and network connection behavior features are obtained through statistical analysis of collected data, collectively referred to as statistical features. Data flow features can be extracted from slices of the network layer, transport layer, and application layer, usually by first calculating traffic statistics and then extracting malicious traffic features from these statistics.
In addition, the classification of encrypted malicious traffic can also be further subdivided based on industry characteristics, such as the Internet of Things, industrial internet, and vehicle networking, where each industry will perform fine-grained classification based on the traffic involved. In summary, there is no absolute standard or unified rule for classifying encrypted malicious traffic. Regardless of the classification method, it relies on key judgment bases such as the characteristics and behaviors of malicious traffic. As the network continues to evolve and technology progresses, the monitoring and analysis methods for encrypted malicious traffic are becoming increasingly diverse, with the reliability of key indicators such as false positive and false negative rates improving. However, it should also be noted that research in the field of monitoring and analyzing encrypted malicious traffic is still a long way to go, with offense and defense being mutually opposing yet interdependent.
3. Key Identification Technologies
It is crucial to use the right methods to detect malicious traffic in encrypted traffic, where features play a key role. Depending on the path of traffic generation, from the source to the destination, various features are involved, such as packet size, direction, protocol, and classification of traffic (service, application). The analytical methods used include statistics, classification, machine learning, and hybrid methods. In the face of a complex and diverse network environment with various data types and terminal devices, it is necessary to select appropriate analytical methods based on the actual situation. The characteristics of feature analysis can be divided into unidimensional features and multidimensional features. Unidimensional features, as the name suggests, focus on a specific characteristic of the data, while multidimensional features involve multiple characteristics, with the aim of improving identification accuracy.
3.1 Unidimensional Feature Traffic Analysis
3.1.1 Certificate Features
Certificates are widely used in networks and are the first barrier for information exchange, serving as an important means of ensuring network security. Server certificates are files used to verify the identity of the server in the Secure Socket Layer (SSL) protocol, which provides secure support for data transmission between the TCP/IP protocol and various application layer protocols. Currently, certificate issuing authorities categorize certificates into three types based on verification levels: Domain Validation (DV) certificates, Organization Validation (OV) certificates, and Extended Validation (EV) certificates. The corresponding server identity authentication certificates are DV SSL certificates, OV SSL certificates, and EV SSL certificates. Among them, DV SSL certificates are issued to websites with less strict review processes, often available for free; OV SSL certificates are generally charged and undergo reviews for the applying enterprises; and EV SSL certificates are issued based on a globally unified strict identity verification standard, representing the highest security level of SSL certificates in the industry. According to the process of establishing a session between client and server, normal sessions will transmit certificates, while malicious sessions will mostly not transmit certificates or will use certificates to hide malicious activities, posing certain threats and challenges to network security. In line with the direction of this research, the basic idea of identifying encrypted malicious traffic through certificates is to quickly identify malicious encrypted traffic by comparing against a malicious certificate feature database. Specifically, during malicious operations, decryption and trusted certificates are still used. By collecting and summarizing a large number of known malicious traffic certificate features, detailed records are made of their version numbers, names, issuance times, etc., and big data analysis is performed. Common features of malicious certificates include self-signing, long certificate validity periods, and the number of certificate extensions. When detecting malicious traffic, the comparison against the malicious certificate feature database is used to verify whether it is malicious traffic. Additionally, manual feature extraction based on certificate text data is also a hot research direction for identifying malicious certificates.
3.1.2 Packet Features
In addition to the aforementioned certificate features, packet features are also an important technology for identifying encrypted malicious traffic. In traffic, packets represent smaller units. By extracting packet features from encrypted traffic, it is possible to classify and identify the payload content within the encrypted traffic. Statistical features of data units include packet size, arrival time series, and byte distribution, among others. The number of packets during normal communication differs from that during communication with malware. When browsing a webpage, the client typically sends fewer request packets to the server, while the server sends many response packets back to the client. However, with malware, the server only sends a few control commands to the client, while the client sends a large number of packets back to the server. Since the characteristics of packet size are not affected by data encryption, they are very suitable for detecting encrypted traffic. Furthermore, from the perspective of data flow size, the downstream traffic is usually much larger than the upstream traffic, while malicious traffic behaves oppositely. When there is a significant increase in upstream traffic, it is necessary to comprehensively assess whether it is benign traffic growth or a malicious traffic attack based on the network situation. This method is relatively complex and may sometimes require combining external intelligence analysis, but the information within packets is rich and complex, and any increase or change in a specific field may indicate a malicious behavior characteristic. The analytical capabilities of encrypted malicious traffic features based on packets also require more technical means for enhancement.
3.1.3 Protocol Features
To ensure network security, many internet encryption protocols have been established, such as the Transport Layer Security (TLS) protocol, Secure Shell (SSH), and Secure Electronic Transaction (SET) protocols. Among them, the TLS protocol is one of the commonly used encryption communication protocols in the industry, positioned between the transport layer and application layer, ensuring confidentiality and data integrity between two communication applications. The corresponding encrypted traffic based on the TLS protocol has become mainstream in the industry, but while enhancing security, it has also introduced network security risks. Many malicious traffic hide within encrypted traffic using the TLS protocol, posing a significant threat to network and business security. The TLS protocol consists of the handshake protocol, record protocol, change cipher spec protocol, and alert protocol. A TLS handshake process is shown in Figure 1, which mainly includes messages such as client hello, server hello, client_key_exchange, and encrypted_handshake_message. These stages include protocol version negotiation, cipher algorithm negotiation, identity authentication, and the determination of session keys and other information. Currently, versions below TLS 1.3 transmit plaintext during the handshake phase, which has become a target for many attackers. Encrypted malicious traffic typically includes the following three types of features: content features, data flow features, and network connection behavior features. Different characteristics can be identified from traffic. Normal encrypted traffic and encrypted malicious traffic differ significantly in terms of the use of cipher algorithms and key lengths. In terms of cipher algorithm usage, malicious traffic often employs outdated or proven insecure algorithms, such as MD5 and RC4; regarding key length, normal encrypted traffic may use a 256-bit key based on elliptic curves, while malicious traffic might use a 2048-bit key based on RSA. In terms of signature methods, malicious traffic typically uses self-signing, lacking trust. Therefore, by detecting the message information exchanged in the TLS protocol, encrypted malicious traffic can be identified based on its features.
Figure 1: TLS Handshake Process
3.2 Multidimensional Feature Traffic Analysis
With the continuous development of big data, methods such as machine learning and deep learning have gained prominence and widespread application. The use of automated identification technology can greatly enhance the efficiency and convenience of traffic identification, while effect evaluation indicators have become important metrics for assessing various methods. The main ideas involve model selection, optimization, and the establishment of feature databases, as well as key indicators such as identification efficiency and accuracy, false negative rates, and false positive rates. Understanding the basic characteristics of data streams is a crucial foundation for applying machine learning model algorithms. Figure 2 shows common data stream features, which include various pieces of information such as version numbers, packet header lengths, timestamps, etc. These can all serve as multidimensional features. By utilizing machine learning algorithms to aggregate and analyze various features, a good model can be obtained, and through continuous optimization of the model, good results can be produced. Common data stream features used in machine learning include temporal and spatial features, header features, payload features, and statistical features. Popular methods include Support Vector Machines (SVM), Random Forests (a classifier that trains and predicts samples using multiple trees), Convolutional Neural Networks (CNN), and Boosting algorithms.
Figure 2: Data Stream Features
The recognition methods based on temporal and spatial features commonly utilize CNN, aiming to leverage deep neural networks to learn the temporal and spatial characteristics of raw traffic data. Temporal and spatial features include characteristics such as the time packets arrive and the direction they are transmitted. The recognition methods based on header features are varied, including clustering, CNN, and Random Forests. From the perspective of small-scale datasets, there is not much difference between machine learning and deep learning. However, when facing large-scale datasets, deep learning exhibits significant advantages, aligning with its essence. The recognition methods based on payload features commonly utilize CNN and SVM. Payload features are complex, mainly including the payload portion of traffic packets. For instance, traffic data can be transformed into visual images, and CNN can be used to classify these images. This method enables end-to-end identification of malicious traffic while meeting the practical application accuracy. Additionally, some methods extract contextual features from the metadata itself; others utilize natural language processing to detect the semantic content of network traffic for malicious application detection.
Recognition methods based on statistical features commonly utilize Random Forests and C4.5 (an algorithm developed by Ross Quinlan for generating decision trees). The C4.5 algorithm is widely applied; for example, the length of encrypted Voice over Internet Protocol (VoIP) packets can be used to identify phrases spoken during calls. Furthermore, the C4.5 algorithm can analyze six statistical features of TLS (upload bytes, download bytes, etc.) and four statistical features in HTTPS streams (user agent, request URL, etc.) to identify malicious application traffic.
In addition to machine learning and deep learning, in recent years, ensemble learning has become a popular method in the field of big data analysis. It is a type of machine learning but is not a standalone machine learning algorithm. Instead, it completes learning tasks by constructing and combining multiple learning machines. The representative of ensemble learning is the Boosting algorithm, which first trains a base learner from the training dataset, then adjusts the training sample distribution based on the performance of the base learning, focusing on the training samples that were misidentified in the previous base learner and adjusting them in the subsequent training process. When the next training iteration begins, a new sample dataset is used to train the next base learner, and the training process ends when the number of base learners reaches a predetermined value. Finally, the predicted result is a weighted combination of the predictions of all base learners. Typical representatives of this algorithm include Gradient Boosting Decision Tree (GBDT), eXtreme Gradient Boosting (XGBoost), and the distributed gradient boosting framework based on decision tree algorithms (LightGBM).
In summary, regardless of the machine learning method used, the core idea revolves around features. Through feature extraction, algorithm model establishment, and tuning, effective analytical results can be produced, and the results can be evaluated. When a single method cannot meet the demands of analyzing traffic data in complex environments, hybrid analytical methods need to be employed. The traffic analysis process based on hybrid methods is shown in Figure 3. Traffic collection is mainly carried out by mirroring or splitting methods to capture traffic at the exit; traffic cleaning and preprocessing convert the traffic into a format suitable for algorithm processing, while also eliminating some invalid data streams to improve the quality of the dataset. The malicious feature identification and analysis is the core component of building the analysis model, selecting appropriate multiple algorithms for malicious traffic identification based on different sample features, and finally outputting the analysis results. The analysis results will also assist in continuously optimizing the model algorithms, further enhancing various evaluation indicators.
Figure 3: Traffic Analysis Process Based on Hybrid Methods
4. Cutting-Edge Technologies
With the continuous enhancement of network security technology capabilities and the integration and innovation of new technologies, the capabilities of analyzing encrypted malicious traffic in the context of big data are also significantly improving. Although some existing technical means can effectively handle this type of malicious traffic, the system of technical means needs continuous improvement, and the precision of malicious traffic feature identification needs further enhancement. Currently, cutting-edge technologies include cryptography, AI, hacker profiling, etc. The integration of new technologies in traffic detection has greatly enhanced detection capabilities.
4.1 Cryptography-Based Feature Analysis
Using cryptography to analyze encrypted malicious traffic is currently a key research direction in both industry and academia. Analyzing encrypted malicious traffic based on cryptography is challenging but also has foresight and reliability. Encrypted traffic itself involves cryptographic techniques, and by analyzing the characteristics of cryptographic techniques used in malicious traffic and comparing them with normal traffic, it is possible to effectively identify malicious traffic. Therefore, researching key technologies in cryptography is necessary. Among these, ciphertext retrieval and ciphertext computation based on public key cryptosystems are the main research hotspots. For example, ciphertext retrieval can directly access ciphertext data through keyword retrieval methods, identifying malicious traffic keywords through single keywords, multiple keywords, fuzzy keywords, and range retrieval.Ciphertext computation involves arbitrary computations on ciphertext data, with its core including homomorphic encryption and secure multi-party computation, primarily achieving secure access and processing of ciphertext. By combining ciphertext retrieval technology, malicious traffic can be identified by searching for malicious keywords in encrypted traffic while protecting user data privacy, thereby exposing malicious traffic.
4.2 AI-Based Feature Analysis
To some extent, using unidimensional and multidimensional feature methods can identify malicious traffic within encrypted traffic. However, these methods lack a global perspective and deeper technical analysis that combines external resources such as threat intelligence and user behavior analysis, making it difficult to maximize the extraction of malicious traffic within encrypted traffic. With the development of Artificial Intelligence (AI) technology, through extensive testing and validation, AI-based security detection of encrypted traffic will become a new technical means. This technical approach empowers malicious traffic detection through AI modeling, parsing, and detection. Due to the flexibility and efficiency of AI, detection effectiveness has significantly improved, demonstrating the high feasibility and good application prospects of AI-based encrypted malicious traffic detection. For instance, based on an AI engine, real-time analysis of all network traffic, combined with threat intelligence data and network behavior analysis techniques, can deeply detect suspicious behaviors, helping to clearly understand the attack chain stage and success probability of attackers.AI-based encrypted malicious traffic analysis is an important direction for future development, integrating AI technology with existing networks. For example, extracting contextual information from TLS/SSL data streams, where the Domain Name System (DNS) plays a crucial role in TLS/SSL communication, can improve the accuracy of AI models by extracting statistical features from DNS contexts. Additionally, usable contexts also include HTTP contexts. Through continuous exploration, it is believed that AI-based detection of encrypted malicious traffic will become increasingly timely and effective.
4.3 Hacker Profiling-Based Feature Analysis
Currently, most technical means are strategic in nature. In the context of cybersecurity offense and defense, talent is the most critical and important factor. User profiling emphasizes different aspects in various fields. For instance, in finance, it focuses on analyzing consumer financial management characteristics, while in e-commerce, it emphasizes consumer spending habits. User profiling technology plays an irreplaceable role in network security areas such as network early warning and traceability, and as its applications expand, it has become an important auxiliary technology for network security protection. In line with this research, constructing a model for analyzing encrypted malicious traffic based on hacker profiling can provide a good early warning effect in advance and assist in traceability and positioning afterward, enriching the malicious feature database. The core idea is to extract hacker profiling features based on analyzing existing hacker attack preferences (attack IP, time period, attack methods, etc.) and to participate in calculations and output analysis results using machine learning, deep learning, and other methods, combining hacker profiling with encrypted traffic data features. By employing hacker profiling-based feature analysis, a warning mechanism for encrypted traffic analysis can be constructed to enhance the capability of defense, achieving timely warnings and tracking of encrypted malicious traffic. Based on the analysis results, the malicious feature database can be continuously enriched, forming a closed-loop effect that effectively blocks malicious attacks and enhances network resilience.
5. Conclusion
This article summarizes the current status of techniques for analyzing and identifying encrypted malicious traffic, focusing on big data analysis methods and cutting-edge technologies. As the legal and regulatory framework related to network security continues to improve, network traffic has become an important data resource in today’s society. Every owner and processor of these traffic resources is a guardian of their security. Technology is continuously advancing, and science and technology are evolving rapidly. It is believed that in the future development of network security, the detection and analysis of encrypted malicious traffic will no longer be a stumbling block in security development, and intelligent, reliable, and diverse technologies will serve as a cornerstone for traffic security protection.
Source: Journal of Information Security and Communication Confidentiality“Submission Contact: 010-82992251 [email protected]”
