A Bug Related to TAG Numbers in Xilinx V7 690T PCIe IP Core Usage

1. Core Background: The “Extended Tag Field” Option of PCIe IP Core

– When instantiating the Xilinx V7 series PCIe Endpoint (EP) IP core in the Vivado tool, you will encounter the configuration option “Extended Tag Field”, which is directly related to the bit width of the TAG number:

– If this option is checked: The IP core claims to support an “8-bit wide TAG field”, theoretically capable of handling 2^8=256 TAG numbers (corresponding to Bit 8 of the Device Control register in the PCIe device space register being set to 1, as clearly defined in Document Table 4-5);

– If this option is not checked: The default uses a “5-bit wide TAG field”, theoretically supporting

• <span>2^5=32 TAG numbers</span>. If unchecked: 5-bit TAG field → theoretically supports 32 TAGs (default).

– The Role of TAG Numbers: A critical “transaction identifier” in the PCIe bus—used to match “initiated requests (such as memory reads, I/O read/writes)” with “completion packets returned by the target (Completion)”, preventing transaction confusion, especially essential in parallel multi-transaction scenarios.

When instantiating the PCIe EP IP core in Vivado, there is an option as follows: Extended Tag Field.If checked, the TAG field is 8 bits wide, supporting 256 TAGs, as shown in the following figure (refer to the IP core manual,PCIe device space register):

sudo lspci -vvv -d:10ee:9037 to check dev ctrl

If, during the instantiation of this IP core, theExtended Tag Field option is accidentally checked, it means that the TAG field is 8 bits wide, supporting 256 TAG numbers, but in reality, the 690T FPGA does not support 256 TAG numbers,and practical results show that if more than 64 TAG numbers are used, the PCIe will hang! (for example, when DMA is processing a very long packet).In other words, regardless of whether you check theExtended Tag Field option, it only supports a maximum of 64 TAG numbers! (Everyone using the Xilinx V7 series FPGA PCIe IP core must pay attention!!!)

2. The Core of the Bug: Theoretical Support vs. Hardware Actual Limitations

– The “theoretical capability” of the IP core configuration: After checking the “Extended Tag Field”, the configuration layer shows support for 256 TAGs;

– The “hardware actual capability” of the 690T FPGA: Practical tests prove that the PCIe integrated block of this FPGA does not support 256 TAGs, and can only stably support a maximum of 64 TAGs.

– Consequences triggered:If the number of parallel PCIe transactions exceeds 64 (i.e., the usage of TAG numbers exceeds 64) during actual use (for example, during long DMA transfers), it will directly lead to PCIe bus hang, and the entire PCIe link will not function properly.

– Key Conclusion:Regardless of whether the “Extended Tag Field” option is checked, the Xilinx V7 690T PCIe IP corecan actually support a maximum of only 64 TAG numbers— the “256 supported” in the configuration option is a “paper parameter” of the IP core, conflicting with the physical limitations of the hardware.

3. Evidence Supporting the Bug

1. Clear Statement in the Official IP Core Manual (PG023)

The official Xilinx manual “Gen3 Integrated Block for PCIe v4.3” (PG023) clearly mentions in the section on “TAG Management for Non-Posted Transactions”:

• The PCIe integrated block’s “requestor” maintains a “Split Completion Table” specifically to store the status of incomplete Non-Posted transactions (such as memory reads, configuration reads/writes);
• The hardware capacity of this table isonly 64 Non-Posted transactions, and the returned completion packets (Completion) are matched with requests through a “6-bit TAG” (<span><span>2^6=64</span></span>)— directly proving that the hardware can only support 64 parallel transactions,which contradicts the theoretical value of 256 TAGs.

Translate again:

2. Actual Debugging Verification of the Open Source Project Corundum

Corundum is an open-source network card project, and its code explicitly limits the configuration for V7 series FPGAs (such as the NetFPGA_SUME platform) to:

This indicates that the project author (Alex Forencich) has discovered the hardware limitations of the V7 690T during actual debugging and specifically limited the number of TAGs to 64 to avoid PCIe hangs.

4. Explanation of Device Control Register Fields

• Bit0: Correctable Error Reporting Enable,this bit is read/write, reset value is 0. When this bit is 1, the PCIe device can send ERR_COR Messages; when 0, it is not supported.

• Bit1: Non-Fatal Error Reporting Enable,this bit is read/write, reset value is 0. When this bit is 1, the PCIe device can send ERR_NON-FATAL Messages; when 0, it is not supported.

• Bit2: Fatal Error Reporting Enable,this bit is read/write, reset value is 0. When this bit is 1, the PCIe device can send ERR_FATAL Messages; when 0, it is not supported.

• Bit3: Unsupported Request Reporting Enable,this bit is read/write, reset value is 0. When this bit is 1, the PCIe device can send Unsupported Requests Error Messages; when 0, it is not supported.

• Bit4: Enable Relaxed Ordering,when this bit is 1, it enables Relaxed Order mode, and the TLP’s Attr field can be set to Relaxed Ordering; when 0, it cannot be set. Reset value is 1, read/write.

• Bit5-7: Max_Payload_Size,read/write, set the maximum TLP payload according to the Device Capability register Bit [2:0]. The system software confirms the value, which cannot exceed the “Max_Payload_Size Supported” field. The maximum payload sent in TLP cannot exceed this value, and TLP received must be less than this value; otherwise, it is considered an error.

• Bit8: Extended Tag Field Enable (core related option),when this bit is 1, the sender can use an 8-bit Tag field; when 0, a 5-bit Tag field is used. Reset value is 1, read/write.

• Bit9: Phantom Functions Enable,when this bit is 1, it enables Phantom Function functionality; when 0, it is not enabled. Reset value is 0, read/write.

• Bit10: Auxiliary (AUX) Power PM Enable,when this bit is 1, the PCIe device can use the auxiliary power provided by the bus.

5. Terminology Supplement

• Non-Posted Transactions:Transactions in the PCIe bus that require “request – response” (such as read operations, configuration operations), which must wait for the target to return the completion packet to be considered complete, requiring TAG number matching;

• Split Completion Table:A table that stores the “incomplete request status” at the hardware level, with capacity determining the maximum number of parallel transactions;

• DMA:Direct Memory Access, commonly used for high-speed data transfers (such as network cards, storage devices), which can generate a large number of parallel PCIe transactions, easily triggering the TAG number overflow bug.

6. Usage Recommendations

If you are using the Xilinx V7 series FPGA (especially the 690T model) PCIe EP IP core, please pay attention to:

• Strict Limitation: Regardless of whether the “Extended Tag Field” option is checked, the actual design must strictly limit the number of parallel PCIe transactions (i.e., TAG number usage) to 64 or fewer.
• Avoid Scenarios: Avoid designing scenarios with “more than 64 parallel Non-Posted transactions” (such as long DMA transfers, multi-port high-speed concurrent access), otherwise it will lead to PCIe link hangs;
• Code Reference: Refer to the open-source project Corundum’s approach, directly setting TAG-related parameters (such as PCIE_TAG_COUNT) to 64 in the code to mitigate risks from the design level.