Recommended by Professor Bao Yungang:
Sharing an innovative work jointly published by the Xiangshan team at the top architecture conference MICRO 2025: DiffTest-H
(1) Origin
DiffTest, originating from the field of software engineering, stands for Differential Test. Its core idea is that for two correct implementations based on a unified specification, given the same defined input, their behaviors should remain consistent; if they are inconsistent, it indicates that at least one implementation is incorrect.
In simple terms, if two people solve the same problem and get the same answer, both pass; if the answers differ, it indicates that someone must be wrong, and a careful check is needed to find out where the mistake occurred.
DiffTest was originally designed by Dr. Yu Zihao in 2017 as a chip functional verification tool for teaching experiments, later applied in the first phase of the “One Life, One Chip” program, playing an irreplaceable role in ensuring successful tape-out.
Subsequently, DiffTest was applied in the development of Xiangshan, becoming a powerful tool for system-level functional verification and an important component of the paper published by the Xiangshan team at MICRO 2022 (selected as one of the 2022 IEEE MICRO Top Picks).
(2) Challenges
In practice, DiffTest connects two different RISC-V processor implementations for differential testing: one is a coarse-grained C language version that can run at hundreds of MHz, while the other is an RTL implementation that is very precise but has a simulation speed of only a few kHz.
This means that for the same RISC-V instruction set specification, there is a speed difference of 4-5 orders of magnitude between the C language version and the RTL version! Therefore, how to accelerate RTL simulation becomes crucial.
In general, there are two approaches to accelerate RTL simulation: one is to run RTL simulation software on a large-scale machine, such as using dozens, hundreds, or even thousands of servers (thousands of CPU cores); although each CPU core runs the RTL simulation software at only a few kHz, large-scale parallelism can achieve acceleration, but the time for a single simulation task is not shortened. The other approach is to use hardware simulation accelerators or FPGAs to accelerate RTL simulation, which can often achieve speeds of several MHz or even tens of MHz, significantly reducing the time for individual tasks.
However, while hardware simulation accelerators are fast, the DiffTest framework cannot utilize them, leading to only a message indicating “an error occurred” without being able to pinpoint “where the error is”.
(3) Breaking the Deadlock
In 2022, Shi Kan returned to China from Intel UK to join the Computing Institute, aiming to tackle the aforementioned challenges. With a long-standing technical accumulation in the FPGA field, Professor Shi Kan quickly led the team to build an FPGA-based ENCORE verification platform from scratch, initially implementing the DiffTest framework on FPGA, significantly improving verification efficiency.
However, FPGA resources are limited and cannot accommodate the DiffTest framework needed to support high-performance CPU design verification like Xiangshan, which requires both the C language version and the RTL version to be implemented simultaneously.
Thus, the Xiangshan team proposed an innovative idea, DiffTest-H: running the C language version on an Intel or AMD machine with simulation performance of hundreds of MHz, while the RTL version runs on Palladium or FPGA, theoretically achieving speeds of several MHz or even tens of MHz. If the overall performance of DiffTest can be improved from the original few kHz to several MHz, that would be a three-order-of-magnitude performance improvement, greatly enhancing the verification efficiency of Xiangshan.
(4) Challenges
Running the Xiangshan C language version (i.e., NEMU) on an x86 machine and the RTL code on Palladium or FPGA sounds simple, but implementing DiffTest-H turned out to be far from straightforward—because the DiffTest-H framework requires data synchronization after each instruction execution to compare whether the execution results of the C language version and the RTL version are consistent, this leads to significant communication overhead between the x86 machine and Palladium/FPGA. As a result, the overall performance of DiffTest only improved by 2.5 to 20 times, falling short of the expected three orders of magnitude.
How to overcome this technical challenge? The Xiangshan team modeled the communication between x86 and FPGA, identifying three types of semantic information contained in the communication: structural semantics, sequential semantics, and behavioral semantics. Based on this understanding, they proposed a semantic-aware communication scheme that significantly improved communication efficiency, achieving approximately 80 times improvement over the initial performance, restoring the three orders of magnitude improvement compared to pure software simulation.
(5) Results
In the past six months, DiffTest-H has helped the “Xiangshan” processor identify and locate over 150 complex bugs, covering various categories such as exception interrupt handling errors, cache coherence errors, and vector control logic errors, resulting in over 19 PRs and more than 780 lines of code modifications.
Now, DiffTest-H has been open-sourced (https://github.com/OpenXiangShan/difftest) and applied in the actual development process of the open-source high-performance processor “Xiangshan”, and received all three badges (available, functionally correct, reproducible) during the MICRO paper reproduction (Artifact Evaluation) phase.
In the future, DiffTest-H will continue to advance the exploration of agile verification for processors, enhancing the verification efficiency of processor cores.
Co-simulation is an important means of functional verification for processor cores, providing architectural-level functional verification and instruction-level error debugging granularity, widely used in academia and industry. However, the existing simulation speed still cannot meet the needs of large-scale verification. Hardware simulation platform-based co-simulation acceleration methods have a higher theoretical speed limit, but frequent software-hardware communication becomes a critical bottleneck for performance.
The Institute of Computing Technology, Chinese Academy of Sciences (referred to as “Computing Institute”) and the Beijing Open Source Chip Research Institute (referred to as “Open Chip Institute”) published a paper titled “DiffTest-H: Toward Semantic-Aware Communication in Hardware-Accelerated Processor Verification” at MICRO 2025,proposing a semantic-aware communication optimization method for hardware-accelerated co-simulation, which reduces communication frequency and data volume while retaining instruction-level error debugging capabilities. DiffTest-H achieves a co-simulation speed of 7.8 MHz on FPGA, helping the “Xiangshan” processor discover over 150 complex bugs and effectively improving the verification efficiency of the “Xiangshan” processor.
Research Background
In the process of processor development, verification has become the most time-consuming key link. To improve verification efficiency, co-simulation runs the design under test (DUT) and the reference model (REF) simultaneously, comparing their architectural states after each instruction execution, providing architectural-level functional verification and instruction-level error debugging granularity, widely used in academia and industry.
In co-simulation, the simulation framework extracts architectural verification events from the DUT for each instruction, such as instruction submission and register updates, driving the REF to execute accordingly and perform comparisons. At the same time, external interrupts and other non-deterministic verification events unique to the DUT can be completely synchronized from the DUT to the REF to align their architectural states.
However, existing co-simulation still faces the problem of simulation speed: traditional software-based RTL simulation methods are limited by large-scale circuit simulation, often only achieving KHz-level simulation speeds; methods based on hardware accelerators (such as Palladium, FPGA) require deploying the DUT and REF on hardware and software sides, respectively, transmitting verification events through software-hardware pathways,as shown in Figure 1. Experimental evaluations indicate that although existing hardware acceleration methods can accelerate DUT circuit simulation by 300 to 10,000 times, the overall co-simulation acceleration ratio is still only 2.5 to 20 times, with up to 99% of simulation time consumed in software-hardware communication. Optimizing software-hardware communication becomes the key issue for improving co-simulation speed.
Figure 1: Co-simulation framework based on hardware simulation platform
Communication Modeling Analysis
In the co-simulation framework based on hardware simulation platforms, software-hardware communication mainly consists of three stages: communication initiation, data transmission, and software processing. Each verification event involves a communication, transmitting the DUT architectural state to the REF for comparison. Taking the DiffTest co-simulation verification framework as an example, it covers 32 types of verification events, generating an average of about 15 communications and 1.2KB of data transmission per cycle, resulting in high software-hardware communication overhead.
Software-hardware communication overhead can be characterized by the following formula:
Where, communication initiation delay includes the data preparation and software-hardware synchronization time before each communication, such as performing valid-ready handshakes between software and hardware and packaging hardware signals into PCIe format data packets. This overhead is related to communication frequency; frequent short data transmissions will generate significant communication initiation delays; data transmission overhead represents the transmission delay of verification events through limited bandwidth pathways, related to data volume and software-hardware pathway bandwidth; software processing delay represents the time taken for the software side to run the reference model and perform co-simulation checks, where existing serial execution methods require hardware to pause and wait for software to finish processing.
In different designs under test (NutShell, Xiangshan) and different hardware simulation platforms (Palladium, FPGA), the proportions of different communication stages’ overhead are shown in Figure 2. The results indicate that for co-simulation scenarios with high communication frequency and low single communication data volume, communication initiation delay becomes the primary component of communication overhead. Additionally, in more complex processor verification scenarios (such as “Xiangshan” processor verification) with larger verification state types and data volumes, data transmission overhead significantly increases; in high-latency, high-bandwidth software-hardware pathways (such as FPGA PCIe), the proportion of communication initiation delay becomes more pronounced.
Figure 2: Breakdown of software-hardware communication overhead for different designs under test and simulation platforms
Existing software-hardware communication optimization work focuses on the three stages of communication:
▲
Communication initiation stage: Packaging different types of verification events to reduce communication frequency, for example, packaging all verification events within a cycle into a long data packet, performing a single software-hardware handshake overall;
▲
Data transmission stage: Compressing the same type of verification events to reduce communication data volume, for example, N instruction submissions can be compressed into a single instruction submission with N instructions;
▲
Software processing stage: Utilizing software-hardware parallelism to mask software processing delays, for example, using non-blocking transmission and comparison methods, allowing hardware to continue running while software is processing.
However, existing work has not completely eliminated the software-hardware communication speed bottleneck. Existing SOTA work Fromajo achieves only about 1 MHz co-simulation speed on a 100 MHz FPGA platform. Moreover, compression loses the architectural behavior details of individual instructions, leading to a loss of debugging capability, making it difficult to pinpoint specific error causes when errors occur.
Semantic-Aware Communication
To address the communication optimization challenges, Shannon-Weaver proposed the concept of “semantic communication”, emphasizing that “understanding the semantics of information” can significantly enhance communication efficiency.In co-simulation, verification events also carry three types of key semantic attributes, which can be used to optimize communication::
Structural Semantics: Length and Data Structure of Verification Events
Different event types have significant differences in length and data structure, making it difficult for hardware packaging and software unpacking. For example, the DiffTest co-simulation verification framework covers 32 types of verification events, with length differences of up to 170 times. Existing packaging methods allocate a fixed space for each type of event in the data packet, leading to over 60% of invalid padding and bandwidth waste when verification events are invalid.
By utilizing structural semantics, the hardware side can dynamically allocate space based on the actual length of verification events, tightly packing variable-length events of different categories, and the software side can parse them according to the data structure. Compact packaging can eliminate invalid padding in data packets, improving bandwidth utilization.
Sequential Semantics: Check Order of Verification Events
Different types of verification events follow specific check orders. For example, non-deterministic verification events such as external interrupts will update the REF architectural state, requiring that previous instructions have been checked and subsequent instructions have not been checked. Existing compression methods couple the check order of compressed events with the communication order, and once a non-deterministic event is encountered during instruction compression, it requires terminating instruction compression and transmitting to the software side to ensure that all previous instructions have been checked. This leads to frequent interruptions in compression, reducing the compression rate.
By utilizing sequential semantics, the check order of compressed events can be decoupled from the communication order, allowing non-deterministic events to be transmitted first with a sequence identifier, while other verification events continue to be compressed, and the software side can restore the check order based on the sequence identifier. Decoupling the sequence in compression can reduce interruptions caused by non-deterministic events, lowering the software-hardware communication data volume.
Behavioral Semantics: Architectural Behavior of Verification Event Checks
Each verification event checks the architectural behavior of the corresponding instruction and locates errors to related micro-architectural components. However, compression loses the behavioral details of individual instructions, retaining only the overall behavioral results after compression, making it difficult to pinpoint error causes. Existing debugging methods rely on architectural verification events to locate errors but still depend on micro-architectural waveform debugging, using hardware snapshots to re-simulate the entire DUT, incurring high resource and time costs.
By utilizing behavioral semantics, it is possible to recheck only the verification events near the error location without re-simulating the entire DUT, providing instruction-level architectural behavior details and maintaining instruction-level error debugging capabilities.
DiffTest-H Design
Building on the team’s previous work on the DiffTest co-simulation verification framework (MICRO’22), this paper proposes a semantic-aware hardware-accelerated co-simulation framework, DiffTest-H, which optimizes software-hardware communication in co-simulation while retaining instruction-level debugging granularity.
DiffTest-H mainly includes three design points:
▲
Batch: Optimizing communication frequency, supporting compact packaging of verification data with different structures into a single communication
▲
Squash: Reducing transmission data volume, decoupling compression and checking of verification data to fully enhance compression rate;
▲
Replay: Maintaining instruction-level error debugging granularity, only replaying verification events near error points, utilizing instruction-level architectural behavior details to assist debugging.
DiffTest-H in the dual-coreXiangshanverification framework is illustrated in Figure 3.
Figure 3: Schematic of DiffTest-H verification framework for dual-core Xiangshan
Batch: Compactly packaging different structured verification events
To minimize communication frequency, this work proposes a communication packaging scheme based on structural semantics, with the workflow illustrated in Figure 4.
Batch mainly includes two stages: hardware-side data packaging and software-side unpacking:
▲
In the hardware-side packaging process, Batch uses structured verification information as the basic unit for data packaging, supporting compact concatenation of variable-length data from different structures, eliminating data padding; meanwhile, it generates metadata that records data structure and quantity information, and generates corresponding software-side parsing methods during the compilation phase.
▲
In the software-side unpacking process, Batch calls the corresponding parsing methods based on metadata to extract verification data of specific lengths from the communication data packet and restores the binary bitstream to a verification state with a specific data structure.
Experimental results show that Batch can reduce communication frequency by approximately 43 times.
Figure 4: Workflow illustration of Batch
Squash: Efficient compression decoupling check order
To effectively reduce the communication data volume of co-simulation, this work proposes a data compression scheme based on sequential semantics, which can significantly enhance compression efficiency while retaining the correct check order, with the workflow illustrated in Figure 5.
Squash includes compression scheduling, data differencing, and reordering three stages:
▲
In the compression scheduling stage, Squash supports decoupling the transmission and check order of different compression events to ensure compression continuity: non-deterministic events can carry sequence identifiers, be transmitted first, and cached on the software side; meanwhile, other verification events can continue to be compressed, maximizing compression rate.
▲
In the data differencing stage, Squash further splits compressed events into finer-grained units, such as different fields in CSR. The hardware side performs differencing comparisons on two consecutive submissions of the same type of compressed events, transmitting only the changed parts, thus removing data redundancy. The software side completes the data based on the unchanged parts, restoring the complete structure of the verification event.
▲
In the software-side reordering stage, different categories of compressed events are reordered based on sequence identifiers to restore the correct check order.
Experimental results indicate that Squash can reduce communication data volume by 47 times without affecting the correct check order.
Figure 5: Workflow illustration of Squash
Replay: Instruction-Level Error Debugging
To retain instruction-level error debugging capability, this work proposes an error debugging scheme based on the behavioral semantics of verification events, supporting the retention of architectural behavior details of each instruction while accelerating with Squash compression, with the workflow illustrated in Figure 6.
Replay mainly includes hardware retransmission and software reprocessing:
▲
In the hardware retransmission part, Replay first caches the compressed verification events into a Replay Buffer and adds a tag before compression. This tag will accompany the entire compression process, indicating the range of original events corresponding to the compressed events. When the software side detects an error, this tag will be used to notify the hardware side to determine the range and retransmit the corresponding original uncompressed events.
▲
In the software reprocessing part, Replay needs to restore the state of the reference model to the pre-error state to recheck the retransmitted events. Since each time a compressed event is checked, an error may occur, to reduce the overhead of frequent snapshots, Replay proposes a lightweight state recovery method based on modification logs, saving the original values before each memory access to the log and restoring them in reverse order when an error occurs, thus achieving lightweight state recovery.
Experimental results show that Replay can achieve instruction-level error debugging granularity without affecting the acceleration optimization effects.
Figure 6: Workflow illustration of Replay
Overall Workflow
The workflow of DiffTest-H is illustrated in Figure 7. On the hardware side, verification information probes collect verification events from different categories and cycles (①), caching the original events for possible debugging needs (②). The verification events undergo compression (③) and packaging (④) to obtain uniformly formatted communication data packets, which are transmitted to the software side through non-blocking software-hardware pathways (⑤), masking software processing delays with software-hardware parallelism. On the software side, the packaged events are disassembled, restored to the corresponding data structures, and checked item by item (⑥).
When inconsistencies between the DUT and REF states are detected, the Replay debugging module notifies the hardware side to retransmit the corresponding original verification events (⑦) based on the software error information, while restoring the reference model to the pre-error state, rechecking instruction by instruction, and obtaining the error cause and现场报告 (⑧).
Figure 7: Workflow illustration of DiffTest-H verification
Experimental Results Evaluation
This work evaluates the effects of different optimization combinations of DiffTest-H based on different designs under test and different hardware simulation platforms, as shown in Table 1. The results indicate that DiffTest-H achieves good acceleration ratios of 74 to 80 times across different designs under test and hardware simulation platforms.
Table 1: Optimization breakdown of DiffTest-H on different DUTs and hardware simulation platforms
DiffTest-H supports acceleration co-simulation based on Emulator/FPGA hardware simulation platforms. Compared to existing work, DiffTest-H achieves lower software-hardware communication overhead and higher co-simulation speeds while ensuring more comprehensive verification (more verification state types and single-cycle verification data volume). Among them, the evaluation results based on FPGA improve by more than 7.8 times compared to existing SOTA. At the same time, DiffTest-H retains instruction-level error debugging granularity, enabling better error localization.
DiffTest-H has been open-sourced (https://github.com/OpenXiangShan/difftest) and applied in the actual development process of the open-source high-performance processor “Xiangshan”, and received all three badges (available, functionally correct, reproducible) during the paper reproduction (Artifact Evaluation) phase. In the past six months,DiffTest-H has helped the “Xiangshan” processor discover and locate over 150 complex bugs, covering various categories such as exception interrupt handling errors, cache coherence errors, vector control logic errors, resulting in over 19 PRs and more than 780 lines of code modifications. In the future, DiffTest-H will continue to advance the exploration of agile verification for processors, enhancing the verification efficiency of processor cores.