1
Introduction
1.1 Research Background
With the rapid development of Internet of Things (IoT) technology, the Internet of Vehicles (IoV), as an important branch, is ushering in unprecedented development opportunities. The IoV achieves information exchange and data sharing between vehicles, roads, and cloud platforms through the collaborative work of various devices and systems, including onboard terminals, roadside devices, and cloud platforms. During the operation of IoV systems, massive amounts of data are generated, covering real-time vehicle location information, driving status data (such as speed, acceleration, steering angle, etc.), vehicle fault diagnosis data, traffic condition data, and user interaction data.
According to relevant statistics, an ordinary intelligent connected vehicle can generate data amounts of tens of GB or even hundreds of GB per hour. In the future era of autonomous driving, the data generated by vehicles will show exponential growth. Such a large volume of data provides a solid data foundation for innovative applications in the IoV (such as intelligent traffic scheduling, autonomous driving decision-making, remote vehicle diagnosis and maintenance, etc.), but it also brings a series of challenges in data processing, among which data deduplication is a key issue that needs to be addressed.
During the collection, transmission, and storage of IoV data, data duplication can easily occur due to various factors. For example, in the data collection phase, onboard sensors may collect the same data multiple times due to equipment failures, signal interference, etc.; in the data transmission phase, network delays and packet retransmission mechanisms may cause data to be sent and received multiple times during transmission; in the data storage phase, due to the replication mechanism of distributed storage systems and data backup strategies, data may also be stored redundantly across different storage nodes.
1.2 Research Significance
The issue of data duplication not only wastes a large amount of storage resources and increases data storage costs but also has a serious negative impact on subsequent data processing and analysis work. First, duplicate data occupies a large amount of storage space, and as the data volume continues to grow, the procurement and maintenance costs of storage devices will increase significantly, placing a heavy economic burden on enterprises. Second, during data processing, duplicate data increases the time and complexity of data processing, reducing processing efficiency. For example, in tasks such as data mining and machine learning model training, duplicate data can lead to extended model training times and reduced model accuracy, affecting the accuracy and reliability of data analysis results.
Moreover, duplicate data may also cause failures in IoV applications. For instance, in intelligent traffic scheduling systems, duplicate vehicle location data may lead to erroneous traffic scheduling decisions, causing traffic congestion; in remote vehicle diagnosis and maintenance systems, duplicate fault diagnosis data may lead maintenance personnel to misjudge the cause of faults, affecting the quality and safety of vehicle repairs. Therefore, conducting research on IoV data deduplication has significant practical significance and value.
Bloom filters, as an efficient probabilistic data structure, have advantages such as small storage space and fast query speed, and have been widely used in data deduplication, cache penetration protection, and other fields. Applying Bloom filters to IoV data deduplication is expected to solve issues such as large data volumes, high data duplication rates, and high real-time requirements, improving the efficiency and quality of IoV data processing and providing strong technical support for the healthy development of the IoV.
1.3 Current Research Status
1.3.1 Domestic Research Status
Domestic scholars have conducted a large amount of research in the field of IoV data deduplication. In recent years, with the rapid development of IoV technology, domestic universities, research institutions, and enterprises have increased their research investment in IoV data processing technologies. In terms of data deduplication technology, domestic research mainly focuses on data deduplication based on hash algorithms, clustering algorithms, and probabilistic data structures (such as Bloom filters).
For example, some research teams have proposed IoV data deduplication methods based on improved hash algorithms, enhancing the accuracy and efficiency of data deduplication by optimizing the design of hash functions. Additionally, some scholars have applied clustering algorithms to IoV data deduplication, categorizing similar data through clustering analysis to achieve data deduplication. Furthermore, domestic scholars have begun to focus on the application of Bloom filters in IoV data deduplication, optimizing the parameters of Bloom filters to improve their performance in this context.
1.3.2 International Research Status
International research on IoV data processing technologies started earlier and is relatively mature. In the field of IoV data deduplication, international research mainly focuses on data deduplication based on distributed systems, machine learning, and new probabilistic data structures.
Some international research institutions have proposed IoV data deduplication methods based on distributed hash tables (DHT), leveraging the advantages of distributed systems to achieve efficient deduplication of large-scale IoV data. Additionally, some scholars have applied machine learning algorithms (such as support vector machines, neural networks, etc.) to IoV data deduplication, achieving automatic identification and filtering of duplicate data through trained machine learning models. Moreover, international scholars have conducted in-depth research on the improvements and extensions of Bloom filters, proposing various new variants such as Counting Bloom Filter and Spectral Bloom Filter, and applying them to IoV data deduplication to further enhance performance and flexibility.
1.4 Research Objectives and Content
1.4.1 Research Objectives
This research aims to design a Bloom filter-based IoV data deduplication scheme to address issues such as large data volumes, high duplication rates, and high real-time requirements, achieving efficient deduplication of IoV data, improving the efficiency and quality of data processing, reducing storage costs, and providing high-quality data support for subsequent IoV applications (such as intelligent traffic scheduling and autonomous driving decision-making).
1.4.2 Research Content
To achieve the above research objectives, this study will mainly carry out the following tasks:
Analyze the characteristics of IoV data and the reasons for data duplication, clarifying the needs and challenges of IoV data deduplication.
Conduct in-depth research on the basic principles, working mechanisms, and performance influencing factors of Bloom filters, providing a theoretical basis for the design of the IoV data deduplication scheme.
Design a Bloom filter-based IoV data deduplication scheme, including data collection and preprocessing modules, Bloom filter parameter optimization modules, data deduplication execution modules, and deduplication effect evaluation modules.
Experimentally validate and evaluate the performance of the designed IoV data deduplication scheme, analyzing its performance in terms of deduplication rate, query efficiency, and storage space usage, and comparing it with other commonly used data deduplication methods.
Optimize and improve the Bloom filter-based IoV data deduplication scheme based on issues discovered during experiments, further enhancing its performance and practicality.
2
Characteristics of IoV Data and Deduplication Demand Analysis
2.1 Characteristics of IoV Data
2.1.1 Large Data Volume
As mentioned earlier, there are numerous vehicles in the IoV, each equipped with a large number of sensors (such as GPS sensors, accelerometers, gyroscopes, cameras, radars, etc.), which collect various operational status data and environmental data of vehicles in real-time. It is estimated that a high-level autonomous vehicle can generate data volumes reaching TB levels daily, while an IoV system in a large city can generate data volumes reaching PB levels or even EB levels daily. Such a massive data volume poses significant challenges for data storage, transmission, and processing.
2.1.2 Diverse Data Types
IoV data types are very rich, encompassing structured data, semi-structured data, and unstructured data. Structured data mainly includes basic vehicle information (such as vehicle identification number, model, license plate number, etc.), real-time location information (such as latitude, longitude, altitude, etc.), driving status data (such as speed, acceleration, steering angle, braking status, etc.), and vehicle fault diagnosis data (such as fault codes, fault descriptions, etc.); semi-structured data mainly includes vehicle log data, traffic condition data (such as traffic flow, congestion level, etc.); unstructured data mainly includes image data collected by vehicle cameras, point cloud data collected by radars, and voice interaction data. Different types of data have different characteristics and processing requirements, increasing the difficulty of IoV data deduplication.
2.1.3 High Real-time Requirements
Many applications in the IoV (such as intelligent traffic scheduling, autonomous driving, vehicle collision warning, etc.) have very high real-time requirements for data. For example, in autonomous driving systems, vehicles need to quickly make decisions and control commands based on real-time collected environmental data (such as the positions and speeds of other vehicles, pedestrian information, etc.) to ensure driving safety. If data processing is not timely, delays or interference from duplicate data may lead to erroneous decisions in the autonomous driving system, resulting in serious safety accidents. Therefore, IoV data deduplication schemes must possess high real-time capabilities, capable of quickly processing large amounts of real-time data and promptly filtering out duplicate data.
2.1.4 Strong Data Dynamics
Vehicles in the IoV are in a constant state of motion, and information such as vehicle position, speed, and direction of travel changes over time, while traffic conditions, weather conditions, and other environmental information also change in real-time. This results in IoV data having strong dynamics, with high update frequencies and short data lifecycles. Additionally, the number of devices in the IoV is also constantly changing, with new vehicles joining the IoV system and old vehicles exiting the system, further increasing the dynamics of IoV data. The dynamic nature of data poses challenges for data deduplication, requiring deduplication schemes to adapt to dynamic changes in data and timely update deduplication rules and strategies.
2.1.5 High Reliability Requirements
IoV data is directly related to vehicle driving safety and the normal operation of traffic systems, thus requiring very high reliability. During the collection, transmission, and storage of IoV data, various factors (such as equipment failures, signal interference, network attacks, etc.) may affect the data, leading to loss, damage, or tampering. The presence of duplicate data may also affect data reliability; for example, duplicate fault diagnosis data may lead maintenance personnel to misjudge the cause of faults, affecting the quality and safety of vehicle repairs. Therefore, IoV data deduplication schemes must not only effectively remove duplicate data but also ensure the integrity and accuracy of the data after deduplication.
2.2 Analysis of Causes of Data Duplication in IoV
2.2.1 Data Collection Phase
In the data collection phase, onboard sensors are the main source of IoV data. However, onboard sensors may cause data duplication due to the following reasons:
Sensor failures: Some onboard sensors may experience failures, such as decreased sensitivity or abnormal sampling frequency, leading to the same data being collected multiple times. For example, GPS sensors may collect the same location information multiple times due to unstable signal reception.
Unreasonable sampling frequency settings: To ensure data accuracy and real-time performance, onboard sensors usually set a certain sampling frequency. If the sampling frequency is set too high, it may lead to the collection of a large amount of similar or even identical data in a short period, resulting in data duplication.
Redundant data from multiple sensors: IoV systems typically equip various types of sensors, which may collect the same or similar data. For example, vehicle speed information can be collected by both GPS sensors and wheel speed sensors; if these two types of sensors have the same sampling time and accuracy, duplicate data may occur.
2.2.2 Data Transmission Phase
The data transmission phase is the process of transmitting IoV data from onboard terminals to roadside devices or cloud platforms, during which data duplication may occur due to the following reasons:
Network delays and packet retransmission: IoV data transmission mainly relies on wireless communication networks (such as 4G, 5G, V2X, etc.), which may experience network delays and packet loss. To ensure reliable data transmission, retransmission mechanisms are usually employed. When data is lost during transmission, the sender will resend the data; if the receiver does not correctly identify the duplicate data, it will lead to the reception and storage of duplicate data.
Changes in network topology: Vehicles in the IoV are in a constant state of motion, frequently establishing and breaking communication connections with roadside devices and other vehicles, leading to continuous changes in network topology. During changes in network topology, data transmission paths may change, resulting in duplicate data transmission.
Defects in data transmission protocols: Currently, IoV data transmission mainly uses some general network transmission protocols (such as TCP/IP protocols), which were not fully designed to consider the characteristics and needs of IoV data. For example, while the retransmission mechanism of the TCP protocol can ensure reliable data transmission, it may lead to a large amount of duplicate data transmission under poor network conditions.
2.2.3 Data Storage Phase
In the data storage phase, IoV data is usually stored in distributed storage systems to meet the needs of large-scale data storage. However, during data storage, data duplication may occur due to the following reasons:
Replication mechanisms in distributed storage systems: To improve data reliability and availability, distributed storage systems usually employ replication mechanisms, copying data to multiple storage nodes. If the replication mechanism is poorly designed or errors occur during data replication, it may lead to data being stored redundantly across different storage nodes.
Data backup strategies: IoV systems typically perform regular data backups to prevent data loss. If the data backup strategy is unreasonable, such as having too high backup frequency or too broad backup scope, it may lead to backup data being duplicated with the original data, increasing data storage costs.
Non-unified data storage formats: IoV data types are diverse, and different types of data may use different storage formats. If data storage formats are not unified, it may lead to data parsing errors during data integration and processing, resulting in duplicate data.
2.3 Demand for IoV Data Deduplication
2.3.1 High Deduplication Rate
The primary demand for IoV data deduplication is to achieve a high deduplication rate, effectively removing duplicate records from the data and reducing the impact of duplicate data on subsequent data processing and analysis work. A high deduplication rate can improve data quality and usability, lower data storage costs, and enhance data processing efficiency. In IoV applications, different application scenarios may have varying requirements for deduplication rates. For example, in vehicle fault diagnosis and maintenance systems, a high deduplication rate for fault diagnosis data is required to ensure that maintenance personnel can accurately determine the cause of faults; while in traffic condition monitoring systems, the deduplication rate for traffic flow data may be relatively lower, as long as it meets the basic needs of traffic scheduling.
2.3.2 High Real-time Performance
As mentioned earlier, many applications in the IoV have very high real-time requirements for data, so IoV data deduplication schemes must possess high real-time capabilities, capable of quickly processing large amounts of real-time data and promptly filtering out duplicate data. The real-time performance of data deduplication is mainly reflected in the latency time of data processing, which is the time from data entering the deduplication system to the completion of deduplication and output of results. To meet the real-time requirements of IoV applications, the processing latency time of the data deduplication system should be as short as possible, typically required to be in the millisecond or even microsecond range.
2.3.3 Low Storage Space Occupation
Given the massive volume of IoV data, deduplication schemes should minimize storage space occupation while achieving deduplication functionality. If the deduplication scheme itself requires a large amount of storage space, it will lose its significance in reducing data storage costs. Therefore, when designing IoV data deduplication schemes, data structures and algorithms with small storage space requirements should be chosen to improve storage space utilization.
2.3.4 High Reliability
IoV data deduplication schemes should have high reliability, ensuring normal operation under various complex environments (such as equipment failures, network interruptions, sudden increases in data volume, etc.) without data loss, damage, or deduplication errors. Additionally, the deduplication scheme should possess a certain level of fault tolerance, allowing for quick recovery to normal operation when system failures occur, ensuring the continuity and stability of data deduplication work.
2.3.5 Good Scalability
With the continuous development of IoV technology, the number of vehicles, devices, and data volume in the IoV will continue to increase, so IoV data deduplication schemes should have good scalability, capable of adapting to the growth of data volume and the expansion of system scale. Scalability mainly includes horizontal and vertical expansion. Horizontal expansion refers to increasing the number of server nodes to enhance the system’s processing capacity and storage capacity; vertical expansion refers to improving the hardware configuration of individual server nodes (such as CPU, memory, hard disk, etc.) to enhance system performance.
3
Principles and Performance Analysis of Bloom Filters
3.1 Basic Principles of Bloom Filters
Bloom filters are a probabilistic data structure proposed by Burton Howard Bloom in 1970, used to determine whether an element belongs to a set. The basic idea is to use multiple hash functions to map elements to multiple positions in a binary array (called a bit array) and set these positions to 1. When determining whether an element belongs to the set, the same multiple hash functions are used to map the element to multiple positions in the bit array; if all these positions are 1, the element is considered to possibly belong to the set; if any position is 0, the element is definitely not part of the set.
The structure of a Bloom filter mainly includes the following two parts:
Bit array: The bit array is the core component of the Bloom filter, consisting of an array of 0s and 1s, with the length of the array usually denoted as m. The initial state of the bit array has all elements set to 0.
Hash function set: The hash function set consists of multiple independent hash functions, with the number of hash functions usually denoted as k. Each hash function can map input data of arbitrary length to the index range of the bit array (i.e., between 0 and m-1).
The operations of a Bloom filter mainly include insertion and query operations:
3.1.1 Insertion Operation
The insertion operation is the process of adding an element to the Bloom filter, with the following specific steps:
For the element x to be inserted, use k hash functions h1, h2, …, hk to compute k hash values h1(x), h2(x), …, hk(x).
Take each hash value modulo the length of the bit array m to obtain k index positions i1, i2, …, ik, where ij = hj(x) mod m (j = 1, 2, …, k).
Set the values of the bit array at index positions i1, i2, …, ik to 1, completing the insertion operation for element x.
It is important to note that during the insertion process, if the value at any index position is already 1, there is no need to modify it again; it can remain 1. This is because the state of the Bloom filter’s bit array only records whether an element “may exist,” rather than precisely recording the number of times an element has been inserted, so repeated insertions of the same element do not change the final state of the bit array.
3.1.2 Query Operation
The query operation is the process of determining whether an element belongs to the set represented by the Bloom filter, with the following specific steps:
For the element y to be queried, use the same k hash functions h1, h2, …, hk to compute k hash values h1(y), h2(y), …, hk(y).
Take each hash value modulo the length of the bit array m to obtain k index positions j1, j2, …, jk, where jj = hj(y) mod m (j = 1, 2, …, k).
Check the values of the bit array at index positions j1, j2, …, jk:
If all positions are 1, the element y is considered to “possibly belong” to the set, with a certain probability of misjudgment;
If any position is 0, the element y is “definitely not part of” the set, and this judgment is absolutely accurate.
This “possible existence” versus “absolute non-existence” judgment characteristic is the core feature of Bloom filters as a probabilistic data structure. Its advantage lies in sacrificing some judgment accuracy (allowing for an extremely low misjudgment rate) in exchange for extremely high query speed and minimal storage space occupation, making it very suitable for scenarios like the IoV that have stringent requirements for real-time performance and storage efficiency.
3.2 Analysis of Bloom Filter Misjudgment Rate
The misjudgment rate is a key indicator of Bloom filter performance, referring to the probability of incorrectly judging an element that does not belong to the set as “possibly belonging” to the set, usually denoted as ε. The occurrence of misjudgment is due to hash collisions: when two different elements are mapped by k hash functions to the same set of index positions in the bit array, and these positions have already been set to 1 by a previous element, the subsequent element will be misjudged as existing.
3.2.2 Factors Affecting Misjudgment Rate
From the above formula, it can be seen that the misjudgment rate ε of Bloom filters is mainly influenced by three parameters:
Length of the bit array m: With k and n fixed, a larger m results in a lower misjudgment rate ε. This is because a larger bit array can disperse the hash mapping positions, reducing the probability of hash collisions. However, if m is too large, it will increase storage space occupation, necessitating a trade-off between misjudgment rate and storage costs.
Number of hash functions k: The influence of k on the misjudgment rate shows a “first decrease then increase” trend. When k is small, the coverage of hash mappings is limited, leading to collisions; as k increases, the randomness of mappings enhances, reducing the misjudgment rate; but if k is too large, each element will occupy more positions in the bit array, causing the bit array to fill up quickly and increasing the probability of collisions. By deriving the misjudgment rate formula, it can be obtained:
Number of inserted elements n: With m and k fixed, as n increases, the proportion of 1s in the bit array rises, leading to an increase in the misjudgment rate ε. When n exceeds the design threshold, the misjudgment rate will sharply increase; therefore, Bloom filters typically need to design m and k based on the expected maximum number of elements n.
3.2.3 Misjudgment Rate Control in IoV Scenarios
IoV data deduplication has strict requirements for misjudgment rates: if the misjudgment rate is too high, a large amount of new data may be incorrectly filtered (i.e., “missed judgments”), affecting data integrity; however, a misjudgment rate that is too low (such as close to 0) will lead to a large m, increasing storage and computational overhead, which cannot meet real-time requirements.
3.3 Factors Affecting Bloom Filter Performance
In addition to the misjudgment rate, the performance of Bloom filters is also reflected in query speed, insertion speed, and storage space occupation, which directly affect their applicability in IoV data deduplication.
3.3.1 Query and Insertion Speed
Both the query and insertion operations of Bloom filters require only k hash calculations and k bit array accesses, with a time complexity of O(k), which is independent of the number of inserted elements n. This means that even if the data volume in the IoV grows exponentially, the processing speed of Bloom filters can remain stable, fully meeting the real-time requirements of IoV data (processing in milliseconds or even microseconds).
The key factor affecting query and insertion speed is the efficiency of the hash functions:
If the hash function computation is complex (such as SHA-256 or other cryptographic hashes), it will increase the time for a single hash calculation, reducing overall processing speed;
If lightweight hash functions (such as MurmurHash, FnvHash) are used, it can significantly improve computational efficiency while ensuring hash uniformity. For example, the single computation time of MurmurHash3 is only a few tens of nanoseconds, making it very suitable for high-frequency data processing scenarios in the IoV.
3.3.2 Storage Space Occupation
The storage space occupation of Bloom filters is determined by the length of the bit array m, measured in “bits.” Compared to traditional hash tables (which need to store complete elements or the hash values of elements, measured in “bytes” or “words”), Bloom filters have a significant storage space advantage.
For example, in the IoV, common “vehicle location data” (each data entry includes vehicle ID, timestamp, latitude, and longitude, approximately 32 bytes) is used:
If a hash table is used to store existing data identifiers, storing 100 million entries would require 32 bytes × 100 million = 3.2GB of space;
If a Bloom filter is used (m=2.45×10^9 bits ≈ 306MB), the storage space is only 1/10 of that of the hash table, and as n increases, the storage space difference will further expand.
Additionally, the storage space of Bloom filters can be further optimized through “dynamic expansion” or “segmented storage”: when the number of inserted elements approaches the design threshold n, new Bloom filter instances can be created to insert new data, while old instances are only used for queries, avoiding the sharp increase in misjudgment rates caused by fixed m.
3.3.3 Fault Tolerance
Bloom filters have a read-only structure for the bit array (only supporting modifications from 0 to 1, not from 1 to 0), thus possessing inherent fault tolerance:
If a hash function calculation is incorrect, it will only affect the corresponding bit array position for that hash, not cause the entire filter to fail;
If a small number of bits in the bit array flip (such as due to hardware failures), it will only slightly increase the misjudgment rate, without causing “missed judgments” (i.e., it will not misjudge existing elements as non-existent).
This fault tolerance is particularly important in IoV scenarios: IoV data processing systems may face issues such as onboard terminal hardware failures and fluctuations in roadside device networks, and the high fault tolerance of Bloom filters can ensure the stability of data deduplication work.
3.4 Variants of Bloom Filters and Their Applicability
Traditional Bloom filters have two main limitations: first, they do not support element deletion (it is impossible to reset a 1 in the bit array to 0, as this would lead to a sharp increase in the misjudgment rate); second, they cannot count the number of occurrences of elements. To address these issues, researchers have proposed various Bloom filter variants, some of which can be applied to specific data deduplication scenarios in the IoV.
3.4.1 Counting Bloom Filter (CBF)
Principle: The traditional Bloom filter’s “bit array” is replaced with a “counter array” (each counter is usually 4 or 8 bits), where the values of k counters are incremented by 1 when inserting elements, decremented by 1 when deleting elements, and queried by checking if the values of k counters are all greater than 0.
Advantages: Supports element deletion and counting functions, suitable for scenarios in the IoV where “duplicate data needs to be dynamically updated,” such as:
Temporary retransmission data in IoV data transmission: If data is redundantly received during transmission due to network fluctuations, it can first be stored in CBF, and after confirming that the data has been successfully processed, the data identifier can be deleted from CBF to avoid subsequent duplicate filtering;
Deduplication of short-term trajectory data of vehicles: If it is necessary to filter duplicate location data of vehicles within 10 seconds, old data in CBF can be periodically cleared (by decrementing the counter), ensuring that the filter only acts on recent data.
Limitations: Compared to traditional Bloom filters, CBF increases storage space occupation by k times (for example, a CBF with 4-bit counters occupies 4 times the storage space of a traditional Bloom filter) and has a risk of “counter overflow” (if the same counter is incremented multiple times, it may exceed the counter’s bit limit, leading to data errors).
3.4.2 Spectral Bloom Filter (SBF)
Principle: Based on CBF, optimized through “hash function grouping” and “shared counters” to further improve counting accuracy and space efficiency. SBF divides k hash functions into g groups, each corresponding to a counter array, incrementing only one counter in each group when inserting elements, and improving counting accuracy through collaborative judgment of multiple groups of counters during queries.
Advantages: With the same storage space, SBF’s counting error is much lower than that of CBF, suitable for scenarios in the IoV where “precise counting of duplicate data occurrences” is required, such as:
Deduplication of vehicle fault diagnosis data: By using SBF to count the occurrences of a certain fault code, if the count exceeds a threshold (e.g., 5 times/minute), it is determined to be a real fault, avoiding false alarms triggered by single duplicate data;
Deduplication of traffic flow data: By using SBF to count the number of vehicles passing through a certain section, filtering duplicate vehicle identification data to ensure the accuracy of traffic flow statistics.
3.4.3 Partitioned Bloom Filter (PBF)
Principle: The traditional Bloom filter’s bit array is divided into multiple independent sub-bit arrays (partitions), each corresponding to a hash function, where only one position in each sub-bit array is set to 1 when inserting elements, and similarly for queries.
Advantages:
Reduces the probability of hash collisions: Each sub-bit array is only mapped by one hash function, avoiding frequent access to the same sub-bit array by multiple hash functions, thus reducing collisions;
Supports parallel processing: Different sub-bit arrays can be operated in parallel by different CPU cores or processing nodes, significantly improving the processing speed of massive data in the IoV;
Facilitates dynamic expansion: Depending on the growth of data volume, a specific partition can be expanded without needing to reconstruct the entire filter.
Applicability: Very suitable for distributed data processing scenarios in the IoV, for example: dividing the urban IoV system into multiple subsystems by region, with each subsystem corresponding to a PBF partition, achieving local deduplication of data within the region, and then aggregating through the cloud to reduce cross-regional data transmission.
3.4.4 Variant Selection Recommendations for IoV Scenarios
Based on the different requirements for IoV data deduplication, different Bloom filter variants can be selected:
If only static deduplication is needed (no data deletion, such as historical data cleaning), prioritize the traditional Bloom filter to minimize storage space and maximize processing speed;
If dynamic deduplication is required (temporary duplicate data needs to be deleted, such as real-time transmission data filtering), choose the Counting Bloom Filter and set reasonable counter bit sizes (e.g., 8 bits) to avoid overflow;
If counting occurrences is needed (such as for fault data or traffic data deduplication), choose the Spectral Bloom Filter to improve counting accuracy;
If a distributed processing architecture is adopted (such as multi-regional IoV systems), choose the Partitioned Bloom Filter to support parallel processing and dynamic expansion.
Disclaimer: This article is a review of relevant literature on intelligent networking and information security, disseminating knowledge related to automotive network information security; if there are any related infringement disputes, please contact us in a timely manner for negotiation or deletion.
To obtain the latest research reports and information on network security, please scan the QR code to add customer service WeChat, join the group for free, and receive important reports related to automotive network security.
