In the previous article (Implementing a PCIe Device in Qemu: Implementing Function Level Reset (FLR)), we added the FLR functionality to the PCIe device simulated in Qemu and triggered the reset operation of the device by writing to the PCIe configuration space using windbg. However, the FLR triggered by manually writing to the configuration space registers does not restore the previous configuration after the configuration space registers are reset; for example, the BAR registers will be cleared. If we want the PCIe device to automatically restore the configuration space registers after performing FLR and to function normally, we need to call the reset interface provided by the operating system through the driver. This way, after the device executes FLR, the operating system or driver framework will automatically restore the PCIe configuration space registers. In this article, we will introduce how to obtain the Device Reset Interface provided by the operating system in the PCIe driver and trigger the device’s FLR.
The Windows operating system provides the following interface for hardware drivers to reset devices:
GUID_DEVICE_RESET_INTERFACE_STANDARD
This interface defines a standardized way for drivers to perform reset operations on hardware when a device needs to be reset (for example, when a device malfunctions and needs to be reset to recover). The structure of this interface is defined as follows, and in addition to the basic members that the interface should have, this interface provides a callback function for device reset: DeviceReset. Once we obtain this interface, we can call this callback function to reset the device, and afterward, the PCIe configuration space of the device can also be automatically restored.
The prototype of the reset callback function DeviceReset is as follows, and an important parameter is ResetType, which is an enum with two optional values:FunctionLevelDeviceReset and PlatformLevelDeviceReset.
This callback function also has an optional parameter: ResetParameters. If we choose ResetType as FunctionLevelDeviceReset, this parameter needs to point to a structure that has a DeviceResetCompletion callback function. This function will be called by the operating system’s PCI driver framework after FLR is completed to notify the device driver that FLR has finished.
Next, we will modify the PCIe device driver to support FLR. The first step is to obtain the Device Reset Interface after creating the WDF device object in the driver’s EvtDriverDeviceAdd callback function. The method to obtain the interface is the same as how we previously obtained the Bus Interface, which is by calling the function WdfFdoQueryForInterface, but here we need to use the GUID for the Device Reset Interface.
Obtaining the Device Reset Interface is that simple. The next issue is how to use this interface to perform a reset. Typically, the driver will call the DeviceReset function when it detects that the hardware device has malfunctioned, such as freezing or timing out. However, here we will demonstrate the effect with a simple handling method. The simplest way is to trigger this process through Test IOCTL. We modified the handling of the Opcode for MY_IOCTL_TEST IOCTL to call the DeviceReset function of the Device Reset Interface, encapsulating this operation in the function FunctionLevelReset.
The implementation of the function FunctionLevelReset is straightforward; it simply calls the DeviceReset callback function, specifying the ResetType as FunctionLevelDeviceReset and passing the previously initialized DeviceResetParameters.
Finally, let’s take a look at the DeviceResetCompletion callback function, which simply logs a message. However, in actual situations, we may need to notify the driver to perform recovery operations on the driver’s internal context or retry critical operations when this function is called.
After modifying the driver, let’s verify the effect. Below are the logs output by the driver when installing the driver and sending the MY_IOCTL_TEST request through TestApp, where we can see the address of the DeviceReset callback function obtained from the Device Reset Interface in the logs. The TestApp also triggered the driver to execute the DeviceReset function through the MY_IOCTL_TEST request, and we can finally see that the DeviceResetCompletion callback function was called, indicating that the FLR execution has completed.
Finally, out of curiosity, I checked the name of the DeviceReset callback function in windbg based on the address printed in the logs. I found that it belongs to pci.sys, which is the PCI protocol driver written by Microsoft. This callback function is certainly implemented by Microsoft according to the relevant descriptions of FLR in the PCIe protocol. Once again, I am amazed by the benefits of hardware protocol standardization, where the operating system vendor is responsible for implementing the protocol part (like FLR here), while the hardware vendor is responsible for implementing the driver to control the specific hardware-related logic and integrating their driver into the operating system using the protocol framework provided by the operating system, thus allowing them to utilize the protocol infrastructure provided by the operating system.