1. PCI Bus System Architecture
PCI stands for Peripheral Component Interconnect, which is a universal bus interface standard that has been widely used in current computer systems. PCI provides a complete set of bus interface specifications aimed at describing how to connect peripheral devices in a structured and controlled manner, while also characterizing the electrical characteristics and behavioral protocols of peripheral devices during connection, and detailing how different components in the computer system should correctly interact.
Whether in Intel-based PCs or Alpha-based workstations, PCI is undoubtedly the most widely used bus interface standard today. Unlike the old ISA bus, PCI completely separates the bus subsystem from the storage subsystem within a computer system, and the CPU interacts with the bus subsystem through a device called a PCI bridge, as shown in Figure 1.
Figure 1 PCI Subsystem Architecture
Due to the use of higher clock frequencies, the PCI bus achieves better overall performance than the ISA bus. The clock frequency of the PCI bus typically ranges from 25MHz to 33MHz, with some even reaching 66MHz or 133MHz, and up to 266MHz in 64-bit systems. Although most PCI devices currently use a 32-bit data bus, the PCI specification has provided for a 64-bit extension, allowing the PCI bus to achieve better platform independence. The PCI bus can now be used in architectures such as IA-32, Alpha, PowerPC, SPARC64, and IA-64.
The PCI bus has three significant advantages that enable it to ultimately replace the ISA bus:
-
Better performance in data transmission between the computer and peripherals;
-
Independence from specific platforms;
-
Ease of plug-and-play implementation.
Figure 2 is a typical logical diagram of a computer system based on the PCI bus. The various parts of the system are connected via the PCI bus and PCI-PCI bridges. From the figure, it is easy to see that the CPU and RAM need to connect to PCI bus 0 (the main PCI bus) through the PCI bridge, while graphics cards with PCI interfaces can connect directly to the main PCI bus. The PCI-PCI bridge is a special PCI device responsible for connecting PCI bus 0 and PCI bus 1 (the subordinate PCI bus), with PCI bus 1 typically referred to as the downstream of the PCI-PCI bridge, and PCI bus 0 as the upstream. The devices connected to the subordinate PCI bus are SCSI cards and Ethernet cards. To maintain compatibility with the old ISA bus standard, the PCI bus can also connect to the ISA bus through a PCI-ISA bridge, thus supporting older ISA devices. The ISA bus in the diagram connects to a multifunction I/O controller to control the keyboard, mouse, and floppy drive.
Figure 2 PCI System Diagram
2. Linux Driver Framework
Linux views all external devices as a special type of file, called a “device file”. If system calls act as the interface between the Linux kernel and applications, then device drivers can be seen as the interface between the Linux kernel and external devices. Device drivers abstract the hardware implementation details from applications, allowing applications to operate on external devices as if they were ordinary files.
1. Character Devices and Block Devices
Linux abstracts hardware handling, allowing all hardware devices to be treated like ordinary files: they can utilize the same standard system call interface for operations such as open, close, read, write, and I/O control, and the primary task of the driver is to implement these system call functions. In a Linux system, all hardware devices are represented by a special device file; for example, the first IDE hard drive in the system is represented by /dev/hda. Each device file corresponds to two device numbers: one is the major device number, which identifies the type of device and the driver used; the other is the minor device number, which distinguishes different hardware devices that use the same device driver. The major device number of a device file must match the major device number requested by the device driver when registering the device; otherwise, user processes will be unable to access the device driver.
There are two main types of device files in the Linux operating system: character devices and block devices. Character devices perform I/O operations one byte at a time, and when a read or write request is issued to a character device, the actual hardware I/O occurs immediately. Generally, character devices may or may not have buffers, and they do not support random access. Block devices use a block of system memory as a buffer, and when a user process issues read or write requests to the device, the driver first checks whether the buffer contains the required data; if it does, the corresponding data is returned; otherwise, the appropriate request function is called to perform the actual I/O operation. Block devices are primarily designed for slow devices like disks, aiming to avoid excessive CPU time waiting for operations to complete. Generally speaking, PCI cards typically belong to character devices.
The major device numbers of all registered (i.e., loaded) hardware devices can be obtained from the /proc/devices file. The mknod command can be used to create a specified type of device file while assigning the corresponding major and minor device numbers. For example, the following command:
[root@gary root]# mknod /dev/lp0 c 6 0
will create a character device file /dev/lp0 with a major device number of 6 and a minor device number of 0. When an application makes a system call on a device file, the Linux kernel will call the corresponding driver function based on the device type and major device number of that device file, transitioning from user mode to kernel mode, and then the driver determines the minor device number of the device, ultimately completing the operation on the corresponding hardware.
2. Device Driver Interface
The I/O subsystem in Linux provides a unified standard device interface to other parts of the kernel, which is achieved through the data structure file_operations in include/linux/fs.h:
struct file_operations {
struct module *owner;
loff_t (*llseek) (struct file *, loff_t, int);
ssize_t (*read) (struct file *, char *, size_t, loff_t *);
ssize_t (*write) (struct file *, const char *, size_t, loff_t *);
int (*readdir) (struct file *, void *, filldir_t);
unsigned int (*poll) (struct file *, struct poll_table_struct *);
int (*ioctl) (struct inode *, struct file *, unsigned int, unsigned long);
int (*mmap) (struct file *, struct vm_area_struct *);
int (*open) (struct inode *, struct file *);
int (*flush) (struct file *);
int (*release) (struct inode *, struct file *);
int (*fsync) (struct file *, struct dentry *, int datasync);
int (*fasync) (int, struct file *, int);
int (*lock) (struct file *, int, struct file_lock *);
ssize_t (*readv) (struct file *, const struct iovec *, unsigned long, loff_t *);
ssize_t (*writev) (struct file *, const struct iovec *, unsigned long, loff_t *);
ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int);
unsigned long (*get_unmapped_area)(struct file *, unsigned long,
unsigned long, unsigned long, unsigned long);
};
When an application performs operations such as open, close, read, write on a device file, the Linux kernel accesses the functions provided by the driver through the file_operations structure. For example, when an application performs a read operation on a device file, the kernel will call the read function in the file_operations structure.
3. Device Driver Modules
Device drivers in Linux can be compiled in two ways: one is to compile them directly as part of the kernel, and the other is to compile them as dynamically loadable modules. Compiling into the kernel increases the size of the kernel, requires modification of the kernel source files, and does not allow for dynamic unloading, which is not conducive to debugging. Therefore, it is recommended to use the module approach.
Essentially, modules are also part of the kernel, but they differ from ordinary applications in that they cannot call C or C++ library functions located in user space; they can only call functions provided by the Linux kernel, which can be viewed in /proc/ksyms.
When writing a driver as a module, two essential functions, init_module() and cleanup_module(), must be implemented, and at least the
4. Device Driver Structure
Understanding the basic structure (or framework) of a device driver is very important for developers. Linux device drivers can roughly be divided into the following parts: driver registration and unregistration, device opening and releasing, device read and write operations, device control operations, and device interrupt and polling handling.
-
Driver registration and unregistration: Adding a driver to the system means assigning it a major device number, which can be accomplished by calling register_chrdev() or register_blkdev() during the driver initialization process. When closing a character or block device, unregister_chrdev() or unregister_blkdev() must be called to unregister the device from the kernel and release the occupied major device number.
-
Device opening and releasing: Opening a device is done by calling the open() function in the file_operations structure, which is used by the driver to prepare for future operations. In most drivers, the open() function typically needs to accomplish the following tasks:
-
Check for device-related errors, such as whether the device is ready.
-
If it is the first time opening, initialize the hardware device.
-
Identify the minor device number and update the read/write operation’s current position pointer f_ops if necessary.
-
Allocate and fill in the data structure to be placed in file->private_data.
-
Increment the use count.
Releasing a device is done by calling the release() function in the file_operations structure, which is sometimes also referred to as close(). Its function is exactly the opposite of open(), typically needing to accomplish the following tasks:
-
Decrement the use count.
-
Free the memory allocated in file->private_data.
-
If the use count reaches zero, close the device.
-
Device read and write operations: The read and write operations for character devices are relatively simple and can be performed directly using the read() and write() functions. However, for block devices, the block_read() and block_write() functions need to be called for data read and write operations; these functions will add read and write requests to the device request table so that the Linux kernel can optimize the order of requests. Since operations are performed on memory buffers rather than directly on the device, this significantly speeds up read and write operations. If the required data is not in the memory buffer or if a write operation is needed to send data to the device, the actual data transfer must occur, which is accomplished by calling the request_fn() function in the blk_dev_struct data structure.
-
Device control operations: In addition to read and write operations, applications sometimes need to control the device, which can be accomplished through the ioctl() function in the device driver. The usage of ioctl() is closely related to specific devices, so it needs to be analyzed based on the actual situation of the device.
-
Device interrupt and polling handling: For hardware devices that do not support interrupts, polling is required to check the device status to determine whether to continue data transfer. If the device supports interrupts, operations can be performed in interrupt mode.
3. PCI Driver Implementation
1. Key Data Structures
PCI devices have three types of address spaces: PCI I/O space, PCI memory space, and PCI configuration space. The CPU can access all address spaces on PCI devices, where the I/O space and memory space are available for device driver use, while the configuration space is used by the PCI initialization code in the Linux kernel. The kernel is responsible for initializing all PCI devices at startup, configuring all PCI devices, including interrupt numbers and I/O base addresses, and listing all found PCI devices and their parameters and attributes in the /proc/pci file.
Linux drivers typically use structures to represent a type of device, where the variables in the structure represent a specific device, storing all information related to that device. A good driver should be able to drive multiple identical devices, with each device distinguished by a minor device number. If a structure is used to represent all devices that can be driven by the driver, the minor device number can be simply represented by an array index.
In PCI drivers, the following key data structures play a crucial role:
The pci_driver structure is found in the file include/linux/pci.h, which was added for new PCI device drivers after Linux kernel version 2.4. Its main purpose is to identify devices through the id_table structure, as well as the probe() function for detecting devices and the remove() function for unloading devices:
struct pci_driver {
struct list_head node;
char *name;
const struct pci_device_id *id_table;
int (*probe) (struct pci_dev *dev, const struct pci_device_id *id);
void (*remove) (struct pci_dev *dev);
int (*save_state) (struct pci_dev *dev, u32 state);
int (*suspend)(struct pci_dev *dev, u32 state);
int (*resume) (struct pci_dev *dev);
int (*enable_wake) (struct pci_dev *dev, u32 state, int enable);
};
The pci_dev structure is also found in the file include/linux/pci.h, detailing almost all hardware information of a PCI device, including vendor ID, device ID, various resources, etc.:
struct pci_dev {
struct list_head global_list;
struct list_head bus_list;
struct pci_bus *bus;
struct pci_bus *subordinate;
void *sysdata;
struct proc_dir_entry *procent;
unsigned int devfn;
unsigned short vendor;
unsigned short device;
unsigned short subsystem_vendor;
unsigned short subsystem_device;
unsigned int class;
u8 hdr_type;
u8 rom_base_reg;
struct pci_driver *driver;
void *driver_data;
u64 dma_mask;
u32 current_state;
unsigned short vendor_compatible[DEVICE_COUNT_COMPATIBLE];
unsigned short device_compatible[DEVICE_COUNT_COMPATIBLE];
unsigned int irq;
struct resource resource[DEVICE_COUNT_RESOURCE];
struct resource dma_resource[DEVICE_COUNT_DMA];
struct resource irq_resource[DEVICE_COUNT_IRQ];
char name[80];
char slot_name[8];
int active;
int ro;
unsigned short regs;
int (*prepare)(struct pci_dev *dev);
int (*activate)(struct pci_dev *dev);
int (*deactivate)(struct pci_dev *dev);
};
2. Basic Framework
When implementing a PCI device driver as a module, at least the following parts should be implemented: device module initialization, device open module, data read/write and control module, interrupt handling module, device release module, and device unload module. Below is a typical basic framework of a PCI device driver, from which it is easy to see how these key modules are organized.
/* Specify which PCI devices this driver applies to */
static struct pci_device_id demo_pci_tbl [] __initdata = {
{PCI_VENDOR_ID_DEMO, PCI_DEVICE_ID_DEMO,
PCI_ANY_ID, PCI_ANY_ID, 0, 0, DEMO},
{0,}
};
/* Data structure describing specific PCI devices */
struct demo_card {
unsigned int magic;
/* Linked list to save all similar PCI devices */
struct demo_card *next;
/* ... */
}
/* Interrupt handling module */
static void demo_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
/* ... */
}
/* Device file operation interface */
static struct file_operations demo_fops = {
owner: THIS_MODULE, /* Owner of demo_fops device module */
read: demo_read, /* Read device operation*/
write: demo_write, /* Write device operation*/
ioctl: demo_ioctl, /* Control device operation*/
mmap: demo_mmap, /* Memory remapping operation*/
open: demo_open, /* Open device operation*/
release: demo_release /* Release device operation*/
/* ... */
};
/* Device module information */
static struct pci_driver demo_pci_driver = {
name: demo_MODULE_NAME, /* Device module name */
id_table: demo_pci_tbl, /* List of devices that can be driven */
probe: demo_probe, /* Find and initialize device */
remove: demo_remove /* Unload device module */
/* ... */
};
static int __init demo_init_module (void)
{
/* ... */
}
static void __exit demo_cleanup_module (void)
{
pci_unregister_driver(&demo_pci_driver);
}
/* Entry for loading the driver module */
module_init(demo_init_module);
/* Entry for unloading the driver module */
module_exit(demo_cleanup_module);
The above code provides a typical framework for a PCI device driver, which is a relatively fixed pattern. It is important to note that functions or data structures related to loading and unloading modules should be prefixed with __init, __exit, etc., to distinguish them from ordinary functions. After constructing such a framework, the next task is to complete the various functional modules within the framework.
3. Device Module Initialization
To initialize a PCI device in a Linux system, the following tasks need to be completed:
Check whether the PCI bus is supported by the Linux kernel; check whether the device is plugged into the bus slot, if so, save the information about the occupied slot position. Read the information from the configuration header for use by the driver. When the Linux kernel starts and completes the scanning, logging, and resource allocation for all PCI devices, it establishes the topology of all PCI devices in the system. When the PCI driver needs to initialize a device, it generally calls the following code:
static int __init demo_init_module (void)
{
/* Check if the system supports the PCI bus */
if (!pci_present())
return -ENODEV;
/* Register hardware driver */
if (!pci_register_driver(&demo_pci_driver)) {
pci_unregister_driver(&demo_pci_driver);
return -ENODEV;
}
/* ... */
return 0;
}
The driver first calls the pci_present() function to check whether the PCI bus has been supported by the Linux kernel. If the system supports the PCI bus structure, the return value of this function is 0; if the driver receives a non-zero return value when calling this function, it must abort its task. In kernels prior to 2.4, the pci_find_device() function needed to be manually called to find PCI devices, but in 2.4 and later, a better approach is to call the pci_register_driver() function to register the driver for PCI devices, providing a pci_driver structure where the probe routine specified will be responsible for detecting the hardware.
static int __init demo_probe(struct pci_dev *pci_dev, const struct pci_device_id *pci_id)
{
struct demo_card *card;
/* Start the PCI device */
if (pci_enable_device(pci_dev))
return -EIO;
/* Device DMA identification */
if (pci_set_dma_mask(pci_dev, DEMO_DMA_MASK)) {
return -ENODEV;
}
/* Dynamically allocate memory in kernel space */
if ((card = kmalloc(sizeof(struct demo_card), GFP_KERNEL)) == NULL) {
printk(KERN_ERR "pci_demo: out of memory\n");
return -ENOMEM;
}
memset(card, 0, sizeof(*card));
/* Read PCI configuration information */
card->iobase = pci_resource_start (pci_dev, 1);
card->pci_dev = pci_dev;
card->pci_id = pci_id->device;
card->irq = pci_dev->irq;
card->next = devs;
card->magic = DEMO_CARD_MAGIC;
/* Set to bus master DMA mode */
pci_set_master(pci_dev);
/* Request I/O resources */
request_region(card->iobase, 64, card_names[pci_id->driver_data]);
return 0;
}
4. Opening Device Module
This module mainly implements requesting interrupts, checking read/write modes, and requesting control over the device. When requesting control, if in non-blocking mode, it returns busy; otherwise, the process voluntarily yields, entering a sleep state, waiting for other processes to release control of the device.
static int demo_open(struct inode *inode, struct file *file)
{
/* Request interrupt, register interrupt handler */
request_irq(card->irq, &demo_interrupt, SA_SHIRQ,
card_names[pci_id->driver_data], card)) {
/* Check read/write mode */
if(file->f_mode & FMODE_READ) {
/* ... */
}
if(file->f_mode & FMODE_WRITE) {
/* ... */
}
/* Request control over the device */
down(&card->open_sem);
while(card->open_mode & file->f_mode) {
if (file->f_flags & O_NONBLOCK) {
/* NONBLOCK mode, return -EBUSY */
up(&card->open_sem);
return -EBUSY;
} else {
/* Wait for scheduling, gain control */
card->open_mode |= f_mode & (FMODE_READ | FMODE_WRITE);
up(&card->open_sem);
/* Increment device open count */
MOD_INC_USE_COUNT;
/* ... */
}
}
}
5. Data Read/Write and Control Information Module
The PCI device driver can provide an interface for controlling hardware to applications through the demo_fops structure’s demo_ioctl() function. For example, it can read data from an I/O register and send it to user space:
static int demo_ioctl(struct inode *inode, struct file *file,
unsigned int cmd, unsigned long arg)
{
/* ... */
switch(cmd) {
case DEMO_RDATA:
/* Read 4 bytes of data from the I/O port */
val = inl(card->iobase + 0x10);
/* Transfer the read data to user space */
return 0;
}
/* ... */
}
In fact, the demo_fops structure can also implement operations such as demo_read(), demo_mmap(), etc. The Linux kernel source code in the driver directory provides many source codes for device drivers, where similar examples can be found. In terms of resource access methods, in addition to I/O instructions, there is also access to peripheral I/O memory. Operations on this memory can be done either by remapping the I/O memory to operate as regular memory or by using Bus Master DMA to let the device transfer data to the system memory.
6. Interrupt Handling Module
PC interrupt resources are limited, with only interrupt numbers 0-15 available, so most external devices request interrupt numbers in a shared manner. When an interrupt occurs, the interrupt handler is first responsible for identifying the interrupt and then making further processing.
static void demo_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
struct demo_card *card = (struct demo_card *)dev_id;
u32 status;
spin_lock(&card->lock);
/* Identify interrupt */
status = inl(card->iobase + GLOB_STA);
if(!(status & INT_MASK))
{
spin_unlock(&card->lock);
return; /* not for us */
}
/* Inform device that the interrupt has been received */
outl(status & INT_MASK, card->iobase + GLOB_STA);
spin_unlock(&card->lock);
/* Other further processing, such as updating DMA buffer pointers, etc. */
}
7. Releasing Device Module
The releasing device module is primarily responsible for releasing control over the device, freeing occupied memory and interrupts, and performs tasks that are exactly the opposite of the opening device module:
static int demo_release(struct inode *inode, struct file *file)
{
/* ... */
/* Release control over the device */
card->open_mode &= (FMODE_READ | FMODE_WRITE);
/* Wake up other processes waiting to gain control */
wake_up(&card->open_wait);
up(&card->open_sem);
/* Free interrupt */
free_irq(card->irq, card);
/* Decrement device open count */
MOD_DEC_USE_COUNT;
/* ... */
}
8. Unloading Device Module
Unloading the device module corresponds to initializing the device module and is relatively simple to implement, primarily by calling the pci_unregister_driver() function to unregister the device driver from the Linux kernel:
static void __exit demo_cleanup_module (void)
{
pci_unregister_driver(&demo_pci_driver);
}
4. Loading Process of Linux PCIE Drivers
During hardware power-on initialization, the BIOS firmware uniformly checks all PCI devices and assigns them a non-conflicting address, allowing their drivers to map their registers to these addresses, which are written into each device’s configuration space by the BIOS. Since this activity is a standard PCI activity, it is naturally written into each device’s configuration space rather than their various control register spaces. Of course, only the BIOS can access the configuration space. When the operating system initializes, it allocates a pci_dev structure for each PCI device and reads the addresses obtained and written into the configuration space into the resource field of the pci_dev. Thus, when we read these addresses, we no longer need to access the configuration space, but can directly request them from the pci_dev. The following four functions directly read the relevant data from pci_dev, as defined in include/linux/pci.h:
#define pci_resource_start(dev,bar) ((dev)->resource[(bar)].start)
#define pci_resource_end(dev,bar) ((dev)->resource[(bar)].end)
It should be noted that each PCI device has a total of 6 address spaces from 0-5; we usually only use the first two. Here, we pass parameter 1 to bar, which uses the memory-mapped address space. Regarding the pci_dev structure, let’s also take a look:
/*
* The pci_dev structure is used to describe PCI devices.
*/
struct pci_dev {
struct list_head global_list; /* Node in list of all PCI devices */
struct list_head bus_list; /* Node in per-bus list */
struct pci_bus *bus; /* Bus this device is on */
struct pci_bus *subordinate; /* Bus this device bridges to */
void *sysdata; /* Hook for sys-specific extension */
struct proc_dir_entry *procent;/* Device entry in /proc/bus/pci */
unsigned int devfn; /* Encoded device & function index */
unsigned short vendor;
unsigned short device;
unsigned short subsystem_vendor;
unsigned short subsystem_device;
unsigned int class; /* 3 bytes: (base,sub,prog-if) */
u8 revision;/* PCI revision, low byte of class word */
u8 hdr_type; /* PCI header type (`multi' flag masked out) */
u8 pcie_type; /* PCI-E device/port type */
u8 rom_base_reg; /* Which config register controls the ROM */
u8 pin; /* Which interrupt pin this device uses */
struct pci_driver *driver; /* Which driver has allocated this device */
u64 dma_mask; /* Mask of the bits of bus address this device implements. */
pci_power_t current_state; /* Current operating state. */
pci_channel_state_t error_state; /* Current connectivity state */
struct device dev; /* Generic device interface */
unsigned short vendor_compatible[DEVICE_COUNT_COMPATIBLE];
unsigned short device_compatible[DEVICE_COUNT_COMPATIBLE];
int cfg_size; /* Size of configuration space */
unsigned int irq;
struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */
unsigned int transparent:1; /* Transparent PCI bridge */
unsigned int multifunction:1;/* Part of multi-function device */
unsigned int is_busmaster:1; /* Device is busmaster */
unsigned int no_msi:1; /* Device may not use msi */
unsigned int no_d1d2:1; /* Only allow d0 or d3 */
unsigned int block_ucfg_access:1; /* Userspace config space access is blocked */
unsigned int broken_parity_status:1; /* Device generates false positive parity */
unsigned int msi_enabled:1;
unsigned int msix_enabled:1;
unsigned int is_managed:1;
unsigned int is_pcie:1;
pci_dev_flags_t dev_flags;
atomic_t enable_cnt; /* pci_enable_device has been called */
u32 saved_config_space[16]; /* config space saved at suspend time */
struct hlist_head saved_cap_space;
struct bin_attribute *rom_attr; /* attribute descriptor for sysfs ROM entry */
int rom_attr_enabled; /* has display of the rom attribute been enabled? */
struct bin_attribute *res_attr[DEVICE_COUNT_RESOURCE]; /* sysfs file for resources */
#ifdef CONFIG_PCI_MSI
struct list_head msi_list;
#endif
};
When we know that the system has assigned base addresses to each BAR space, we need to use the ioremap function to map it into the kernel’s virtual address space, making it easier for the kernel to access. The system first allocates the physical address to the PCIE Bar; ioremap is a kernel-provided function used to map peripheral registers to main memory. The addresses we want to map have already been read from pci_dev (in the previous step), making the mapping successful without conflict with other addresses. What effect does mapping have? For example, if a network card has 100 registers, all connected in one block with fixed positions, and if each register occupies 4 bytes, then a total of 400 bytes of space is successfully mapped to memory. The ioaddr is the start of this address (note that ioaddr is a virtual address, while mmio_start is a physical address obtained from the BIOS, which is certainly a physical address, and in protected mode, the CPU recognizes virtual addresses only, not physical addresses). Therefore, ioaddr+0 is the address of the first register, ioaddr+4 is the address of the second register (each register occupies 4 bytes), and so on, allowing us to access all registers in memory and manipulate them. In system function calls, if we want to open a PCIE device, we can use the container_of macro structure to obtain the starting address of the device to be opened.
This article is organized from the internet; if there is any infringement, please delete it.