Linux pci express drivers

2. The PCI Express Port Bus Driver Guide HOWTO¶

В© 2004 Intel Corporation

2.1. About this guide¶

This guide describes the basics of the PCI Express Port Bus driver and provides information on how to enable the service drivers to register/unregister with the PCI Express Port Bus Driver.

2.2. What is the PCI Express Port Bus Driver¶

A PCI Express Port is a logical PCI-PCI Bridge structure. There are two types of PCI Express Port: the Root Port and the Switch Port. The Root Port originates a PCI Express link from a PCI Express Root Complex and the Switch Port connects PCI Express links to internal logical PCI buses. The Switch Port, which has its secondary bus representing the switch’s internal routing logic, is called the switch’s Upstream Port. The switch’s Downstream Port is bridging from switch’s internal routing bus to a bus representing the downstream PCI Express link from the PCI Express Switch.

A PCI Express Port can provide up to four distinct functions, referred to in this document as services, depending on its port type. PCI Express Port’s services include native hotplug support (HP), power management event support (PME), advanced error reporting support (AER), and virtual channel support (VC). These services may be handled by a single complex driver or be individually distributed and handled by corresponding service drivers.

2.3. Why use the PCI Express Port Bus Driver?В¶

In existing Linux kernels, the Linux Device Driver Model allows a physical device to be handled by only a single driver. The PCI Express Port is a PCI-PCI Bridge device with multiple distinct services. To maintain a clean and simple solution each service may have its own software service driver. In this case several service drivers will compete for a single PCI-PCI Bridge device. For example, if the PCI Express Root Port native hotplug service driver is loaded first, it claims a PCI-PCI Bridge Root Port. The kernel therefore does not load other service drivers for that Root Port. In other words, it is impossible to have multiple service drivers load and run on a PCI-PCI Bridge device simultaneously using the current driver model.

To enable multiple service drivers running simultaneously requires having a PCI Express Port Bus driver, which manages all populated PCI Express Ports and distributes all provided service requests to the corresponding service drivers as required. Some key advantages of using the PCI Express Port Bus driver are listed below:

Allow multiple service drivers to run simultaneously on a PCI-PCI Bridge Port device.

Allow service drivers implemented in an independent staged approach.

Allow one service driver to run on multiple PCI-PCI Bridge Port devices.

Manage and distribute resources of a PCI-PCI Bridge Port device to requested service drivers.

2.4. Configuring the PCI Express Port Bus Driver vs. Service Drivers¶

2.4.1. Including the PCI Express Port Bus Driver Support into the Kernel¶

Including the PCI Express Port Bus driver depends on whether the PCI Express support is included in the kernel config. The kernel will automatically include the PCI Express Port Bus driver as a kernel driver when the PCI Express support is enabled in the kernel.

2.4.2. Enabling Service Driver Support¶

PCI device drivers are implemented based on Linux Device Driver Model. All service drivers are PCI device drivers. As discussed above, it is impossible to load any service driver once the kernel has loaded the PCI Express Port Bus Driver. To meet the PCI Express Port Bus Driver Model requires some minimal changes on existing service drivers that imposes no impact on the functionality of existing service drivers.

A service driver is required to use the two APIs shown below to register its service with the PCI Express Port Bus driver (see section 5.2.1 & 5.2.2). It is important that a service driver initializes the pcie_port_service_driver data structure, included in header file /include/linux/pcieport_if.h, before calling these APIs. Failure to do so will result an identity mismatch, which prevents the PCI Express Port Bus driver from loading a service driver.

2.4.2.1. pcie_port_service_register¶

This API replaces the Linux Driver Model’s pci_register_driver API. A service driver should always calls pcie_port_service_register at module init. Note that after service driver being loaded, calls such as pci_enable_device(dev) and pci_set_master(dev) are no longer necessary since these calls are executed by the PCI Port Bus driver.

2.4.2.2. pcie_port_service_unregister¶

pcie_port_service_unregister replaces the Linux Driver Model’s pci_unregister_driver. It’s always called by service driver when a module exits.

2.4.2.3. Sample Code¶

Below is sample service driver code to initialize the port service driver data structure.

Below is a sample code for registering/unregistering a service driver.

2.5. Possible Resource Conflicts¶

Since all service drivers of a PCI-PCI Bridge Port device are allowed to run simultaneously, below lists a few of possible resource conflicts with proposed solutions.

2.5.1. MSI and MSI-X Vector Resource¶

Once MSI or MSI-X interrupts are enabled on a device, it stays in this mode until they are disabled again. Since service drivers of the same PCI-PCI Bridge port share the same physical device, if an individual service driver enables or disables MSI/MSI-X mode it may result unpredictable behavior.

To avoid this situation all service drivers are not permitted to switch interrupt mode on its device. The PCI Express Port Bus driver is responsible for determining the interrupt mode and this should be transparent to service drivers. Service drivers need to know only the vector IRQ assigned to the field irq of struct pcie_device, which is passed in when the PCI Express Port Bus driver probes each service driver. Service drivers should use (struct pcie_device*)dev->irq to call request_irq/free_irq. In addition, the interrupt mode is stored in the field interrupt_mode of struct pcie_device.

2.5.2. PCI Memory/IO Mapped Regions¶

Service drivers for PCI Express Power Management (PME), Advanced Error Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) access PCI configuration space on the PCI Express port. In all cases the registers accessed are independent of each other. This patch assumes that all service drivers will be well behaved and not overwrite other service driver’s configuration settings.

2.5.3. PCI Config Registers¶

Each service driver runs its PCI config operations on its own capability structure except the PCI Express capability structure, in which Root Control register and Device Control register are shared between PME and AER. This patch assumes that all service drivers will be well behaved and not overwrite other service driver’s configuration settings.

© Copyright The kernel development community.

Источник

1. How To Write Linux PCI Drivers¶

Authors:	Martin Mares @ ucw . cz> Grant Grundler @ parisc-linux . org>

The world of PCI is vast and full of (mostly unpleasant) surprises. Since each CPU architecture implements different chip-sets and PCI devices have different requirements (erm, «features»), the result is the PCI support in the Linux kernel is not as trivial as one would wish. This short paper tries to introduce all potential driver authors to Linux APIs for PCI device drivers.

A more complete resource is the third edition of «Linux Device Drivers» by Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman. LDD3 is available for free (under Creative Commons License) from: http://lwn.net/Kernel/LDD3/.

However, keep in mind that all documents are subject to «bit rot». Refer to the source code if things are not working as described here.

1.1. Structure of PCI drivers¶

PCI drivers «discover» PCI devices in a system via pci_register_driver(). Actually, it’s the other way around. When the PCI generic code discovers a new device, the driver with a matching «description» will be notified. Details on this below.

pci_register_driver() leaves most of the probing for devices to the PCI layer and supports online insertion/removal of devices [thus supporting hot-pluggable PCI, CardBus, and Express-Card in a single driver]. pci_register_driver() call requires passing in a table of function pointers and thus dictates the high level structure of a driver.

Once the driver knows about a PCI device and takes ownership, the driver generally needs to perform the following initialization:

• Enable the device
• Request MMIO/IOP resources
• Set the DMA mask size (for both coherent and streaming DMA)
• Allocate and initialize shared control data (pci_allocate_coherent())
• Access device configuration space (if needed)
• Register IRQ handler (request_irq())
• Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip)
• Enable DMA/processing engines

When done using the device, and perhaps the module needs to be unloaded, the driver needs to take the follow steps:

• Disable the device from generating IRQs
• Release the IRQ ( free_irq() )
• Stop all DMA activity
• Release DMA buffers (both streaming and coherent)
• Unregister from other subsystems (e.g. scsi or netdev)
• Release MMIO/IOP resources
• Disable the device

Most of these topics are covered in the following sections. For the rest look at LDD3 or
.

If the PCI subsystem is not configured (CONFIG_PCI is not set), most of the PCI functions described below are defined as inline functions either completely empty or just returning an appropriate error codes to avoid lots of ifdefs in the drivers.

1.2. pci_register_driver() call¶

PCI device drivers call pci_register_driver() during their initialization with a pointer to a structure describing the driver ( struct pci_driver ):

PCI driver structure

Definition

Members

node List of driver structures. name Driver name. id_table Pointer to table of device IDs the driver is interested in. Most drivers should export this table using MODULE_DEVICE_TABLE(pci. ). probe This probing function gets called (during execution of pci_register_driver() for already existing devices or later if a new device gets inserted) for all PCI devices which match the ID table and are not «owned» by the other drivers yet. This function gets passed a «struct pci_dev *» for each device whose entry in the ID table matches the device. The probe function returns zero when the driver chooses to take «ownership» of the device or an error code (negative number) otherwise. The probe function always gets called from process context, so it can sleep. remove The remove() function gets called whenever a device being handled by this driver is removed (either during deregistration of the driver or when it’s manually pulled out of a hot-pluggable slot). The remove function always gets called from process context, so it can sleep. suspend Put device into low power state. suspend_late Put device into low power state. resume_early Wake device from low power state. resume Wake device from low power state. (Please see Documentation/power/pci.rst for descriptions of PCI Power Management and the related functions.) shutdown Hook into reboot_notifier_list (kernel/sys.c). Intended to stop any idling DMA operations. Useful for enabling wake-on-lan (NIC) or changing the power state of a device before reboot. e.g. drivers/net/e100.c. sriov_configure Optional driver callback to allow configuration of number of VFs to enable via sysfs «sriov_numvfs» file. err_handler See Documentation/PCI/pci-error-recovery.rst groups Sysfs attribute groups. driver Driver model structure. dynids List of dynamically added device IDs.

The ID table is an array of struct pci_device_id entries ending with an all-zero entry. Definitions with static const are generally preferred.

PCI device ID structure

Definition

Members

vendor Vendor ID to match (or PCI_ANY_ID) device Device ID to match (or PCI_ANY_ID) subvendor Subsystem vendor ID to match (or PCI_ANY_ID) subdevice Subsystem device ID to match (or PCI_ANY_ID) class Device class, subclass, and «interface» to match. See Appendix D of the PCI Local Bus Spec or include/linux/pci_ids.h for a full list of classes. Most drivers do not need to specify class/class_mask as vendor/device is normally sufficient. class_mask Limit which sub-fields of the class field are compared. See drivers/scsi/sym53c8xx_2/ for example of usage. driver_data Data private to the driver. Most drivers don’t need to use driver_data field. Best practice is to use driver_data as an index into a static list of equivalent device types, instead of using it as a pointer.

Most drivers only need PCI_DEVICE() or PCI_DEVICE_CLASS() to set up a pci_device_id table.

New PCI IDs may be added to a device driver pci_ids table at runtime as shown below:

All fields are passed in as hexadecimal values (no leading 0x). The vendor and device fields are mandatory, the others are optional. Users need pass only as many optional fields as necessary:

• subvendor and subdevice fields default to PCI_ANY_ID (FFFFFFFF)
• class and classmask fields default to 0
• driver_data defaults to 0UL.

Note that driver_data must match the value used by any of the pci_device_id entries defined in the driver. This makes the driver_data field mandatory if all the pci_device_id entries have a non-zero driver_data value.

Once added, the driver probe routine will be invoked for any unclaimed PCI devices listed in its (newly updated) pci_ids list.

When the driver exits, it just calls pci_unregister_driver() and the PCI layer automatically calls the remove hook for all devices handled by the driver.

1.2.1. «Attributes» for driver functions/dataВ¶

Please mark the initialization and cleanup functions where appropriate (the corresponding macros are defined in
):

__init	Initialization code. Thrown away after the driver initializes.
__exit	Exit code. Ignored for non-modular drivers.

1.3. How to find PCI devices manually¶

PCI drivers should have a really good reason for not using the pci_register_driver() interface to search for PCI devices. The main reason PCI devices are controlled by multiple drivers is because one PCI device implements several different HW services. E.g. combined serial/parallel port/floppy controller.

A manual search may be performed using the following constructs:

Searching by vendor and device ID:

Searching by class ID (iterate in a similar way):

Searching by both vendor/device and subsystem vendor/device ID:

You can use the constant PCI_ANY_ID as a wildcard replacement for VENDOR_ID or DEVICE_ID. This allows searching for any device from a specific vendor, for example.

These functions are hotplug-safe. They increment the reference count on the pci_dev that they return. You must eventually (possibly at module unload) decrement the reference count on these devices by calling pci_dev_put() .

1.4. Device Initialization Steps¶

As noted in the introduction, most PCI drivers need the following steps for device initialization:

• Enable the device
• Request MMIO/IOP resources
• Set the DMA mask size (for both coherent and streaming DMA)
• Allocate and initialize shared control data (pci_allocate_coherent())
• Access device configuration space (if needed)
• Register IRQ handler (request_irq())
• Initialize non-PCI (i.e. LAN/SCSI/etc parts of the chip)
• Enable DMA/processing engines.

The driver can access PCI config space registers at any time. (Well, almost. When running BIST, config space can go away. but that will just result in a PCI Bus Master Abort and config reads will return garbage).

1.4.1. Enable the PCI device¶

Before touching any device registers, the driver needs to enable the PCI device by calling pci_enable_device() . This will:

• wake up the device if it was in suspended state,
• allocate I/O and memory regions of the device (if BIOS did not),
• allocate an IRQ (if BIOS did not).

pci_enable_device() can fail! Check the return value.

OS BUG: we don’t check resource allocations before enabling those resources. The sequence would make more sense if we called pci_request_resources() before calling pci_enable_device() . Currently, the device drivers can’t detect the bug when when two devices have been allocated the same range. This is not a common problem and unlikely to get fixed soon.

This has been discussed before but not changed as of 2.6.19: http://lkml.org/lkml/2006/3/2/194

pci_set_master() will enable DMA by setting the bus master bit in the PCI_COMMAND register. It also fixes the latency timer value if it’s set to something bogus by the BIOS. pci_clear_master() will disable DMA by clearing the bus master bit.

If the PCI device can use the PCI Memory-Write-Invalidate transaction, call pci_set_mwi() . This enables the PCI_COMMAND bit for Mem-Wr-Inval and also ensures that the cache line size register is set correctly. Check the return value of pci_set_mwi() as not all architectures or chip-sets may support Memory-Write-Invalidate. Alternatively, if Mem-Wr-Inval would be nice to have but is not required, call pci_try_set_mwi() to have the system do its best effort at enabling Mem-Wr-Inval.

1.4.2. Request MMIO/IOP resources¶

Memory (MMIO), and I/O port addresses should NOT be read directly from the PCI device config space. Use the values in the pci_dev structure as the PCI «bus address» might have been remapped to a «host physical» address by the arch/chip-set specific kernel support.

See Documentation/io-mapping.txt for how to access device registers or device memory.

The device driver needs to call pci_request_region() to verify no other device is already using the same address resource. Conversely, drivers should call pci_release_region() AFTER calling pci_disable_device() . The idea is to prevent two devices colliding on the same address range.

See OS BUG comment above. Currently (2.6.19), The driver can only determine MMIO and IO Port resource availability _after_ calling pci_enable_device() .

Generic flavors of pci_request_region() are request_mem_region() (for MMIO ranges) and request_region() (for IO Port ranges). Use these for address resources that are not described by «normal» PCI BARs.

1.4.3. Set the DMA mask size¶

If anything below doesn’t make sense, please refer to Documentation/DMA-API.txt. This section is just a reminder that drivers need to indicate DMA capabilities of the device and is not an authoritative source for DMA interfaces.

While all drivers should explicitly indicate the DMA capability (e.g. 32 or 64 bit) of the PCI bus master, devices with more than 32-bit bus master capability for streaming data need the driver to «register» this capability by calling pci_set_dma_mask() with appropriate parameters. In general this allows more efficient DMA on systems where System RAM exists above 4G _physical_ address.

Drivers for all PCI-X and PCIe compliant devices must call pci_set_dma_mask() as they are 64-bit DMA devices.

Similarly, drivers must also «register» this capability if the device can directly address «consistent memory» in System RAM above 4G physical address by calling pci_set_consistent_dma_mask(). Again, this includes drivers for all PCI-X and PCIe compliant devices. Many 64-bit «PCI» devices (before PCI-X) and some PCI-X devices are 64-bit DMA capable for payload («streaming») data but not control («consistent») data.

1.4.4. Setup shared control data¶

Once the DMA masks are set, the driver can allocate «consistent» (a.k.a. shared) memory. See Documentation/DMA-API.txt for a full description of the DMA APIs. This section is just a reminder that it needs to be done before enabling DMA on the device.

1.4.5. Initialize device registers¶

Some drivers will need specific «capability» fields programmed or other «vendor specific» register initialized or reset. E.g. clearing pending interrupts.

1.4.6. Register IRQ handler¶

While calling request_irq() is the last step described here, this is often just another intermediate step to initialize a device. This step can often be deferred until the device is opened for use.

All interrupt handlers for IRQ lines should be registered with IRQF_SHARED and use the devid to map IRQs to devices (remember that all PCI IRQ lines can be shared).

request_irq() will associate an interrupt handler and device handle with an interrupt number. Historically interrupt numbers represent IRQ lines which run from the PCI device to the Interrupt controller. With MSI and MSI-X (more below) the interrupt number is a CPU «vector».

request_irq() also enables the interrupt. Make sure the device is quiesced and does not have any interrupts pending before registering the interrupt handler.

MSI and MSI-X are PCI capabilities. Both are «Message Signaled Interrupts» which deliver interrupts to the CPU via a DMA write to a Local APIC. The fundamental difference between MSI and MSI-X is how multiple «vectors» get allocated. MSI requires contiguous blocks of vectors while MSI-X can allocate several individual ones.

MSI capability can be enabled by calling pci_alloc_irq_vectors() with the PCI_IRQ_MSI and/or PCI_IRQ_MSIX flags before calling request_irq(). This causes the PCI support to program CPU vector data into the PCI device capability registers. Many architectures, chip-sets, or BIOSes do NOT support MSI or MSI-X and a call to pci_alloc_irq_vectors with just the PCI_IRQ_MSI and PCI_IRQ_MSIX flags will fail, so try to always specify PCI_IRQ_LEGACY as well.

Drivers that have different interrupt handlers for MSI/MSI-X and legacy INTx should chose the right one based on the msi_enabled and msix_enabled flags in the pci_dev structure after calling pci_alloc_irq_vectors.

There are (at least) two really good reasons for using MSI:

1. MSI is an exclusive interrupt vector by definition. This means the interrupt handler doesn’t have to verify its device caused the interrupt.
2. MSI avoids DMA/IRQ race conditions. DMA to host memory is guaranteed to be visible to the host CPU(s) when the MSI is delivered. This is important for both data coherency and avoiding stale control data. This guarantee allows the driver to omit MMIO reads to flush the DMA stream.

See drivers/infiniband/hw/mthca/ or drivers/net/tg3.c for examples of MSI/MSI-X usage.

1.5. PCI device shutdown¶

When a PCI device driver is being unloaded, most of the following steps need to be performed:

• Disable the device from generating IRQs
• Release the IRQ ( free_irq() )
• Stop all DMA activity
• Release DMA buffers (both streaming and consistent)
• Unregister from other subsystems (e.g. scsi or netdev)
• Disable device from responding to MMIO/IO Port addresses
• Release MMIO/IO Port resource(s)

1.5.1. Stop IRQs on the device¶

How to do this is chip/device specific. If it’s not done, it opens the possibility of a «screaming interrupt» if (and only if) the IRQ is shared with another device.

When the shared IRQ handler is «unhooked», the remaining devices using the same IRQ line will still need the IRQ enabled. Thus if the «unhooked» device asserts IRQ line, the system will respond assuming it was one of the remaining devices asserted the IRQ line. Since none of the other devices will handle the IRQ, the system will «hang» until it decides the IRQ isn’t going to get handled and masks the IRQ (100,000 iterations later). Once the shared IRQ is masked, the remaining devices will stop functioning properly. Not a nice situation.

This is another reason to use MSI or MSI-X if it’s available. MSI and MSI-X are defined to be exclusive interrupts and thus are not susceptible to the «screaming interrupt» problem.

1.5.2. Release the IRQ¶

Once the device is quiesced (no more IRQs), one can call free_irq() . This function will return control once any pending IRQs are handled, «unhook» the drivers IRQ handler from that IRQ, and finally release the IRQ if no one else is using it.

1.5.3. Stop all DMA activity¶

It’s extremely important to stop all DMA operations BEFORE attempting to deallocate DMA control data. Failure to do so can result in memory corruption, hangs, and on some chip-sets a hard crash.

Stopping DMA after stopping the IRQs can avoid races where the IRQ handler might restart DMA engines.

While this step sounds obvious and trivial, several «mature» drivers didn’t get this step right in the past.

1.5.4. Release DMA buffers¶

Once DMA is stopped, clean up streaming DMA first. I.e. unmap data buffers and return buffers to «upstream» owners if there is one.

Then clean up «consistent» buffers which contain the control data.

See Documentation/DMA-API.txt for details on unmapping interfaces.

1.5.5. Unregister from other subsystems¶

Most low level PCI device drivers support some other subsystem like USB, ALSA, SCSI, NetDev, Infiniband, etc. Make sure your driver isn’t losing resources from that other subsystem. If this happens, typically the symptom is an Oops (panic) when the subsystem attempts to call into a driver that has been unloaded.

1.5.6. Disable Device from responding to MMIO/IO Port addresses¶

io_unmap() MMIO or IO Port resources and then call pci_disable_device() . This is the symmetric opposite of pci_enable_device() . Do not access device registers after calling pci_disable_device() .

1.5.7. Release MMIO/IO Port Resource(s)В¶

Call pci_release_region() to mark the MMIO or IO Port range as available. Failure to do so usually results in the inability to reload the driver.

1.6. How to access PCI config space¶

You can use pci_(read|write)_config_(byte|word|dword) to access the config space of a device represented by struct pci_dev * . All these functions return 0 when successful or an error code ( PCIBIOS_. ) which can be translated to a text string by pcibios_strerror. Most drivers expect that accesses to valid PCI devices don’t fail.

If you don’t have a struct pci_dev available, you can call pci_bus_(read|write)_config_(byte|word|dword) to access a given device and function on that bus.

If you access fields in the standard portion of the config header, please use symbolic names of locations and bits declared in
.

If you need to access Extended PCI Capability registers, just call pci_find_capability() for the particular capability and it will find the corresponding register block for you.

1.7. Other interesting functions¶

pci_get_domain_bus_and_slot()	Find pci_dev corresponding to given domain, bus and slot and number. If the device is found, its reference count is increased.
pci_set_power_state()	Set PCI Power Management state (0=D0 . 3=D3)
pci_find_capability()	Find specified capability in device’s capability list.
pci_resource_start()	Returns bus start address for a given PCI region
pci_resource_end()	Returns bus end address for a given PCI region
pci_resource_len()	Returns the byte length of a PCI region
pci_set_drvdata()	Set private driver data pointer for a pci_dev
pci_get_drvdata()	Return private driver data pointer for a pci_dev
pci_set_mwi()	Enable Memory-Write-Invalidate transactions.
pci_clear_mwi()	Disable Memory-Write-Invalidate transactions.

1.8. Miscellaneous hints¶

When displaying PCI device names to the user (for example when a driver wants to tell the user what card has it found), please use pci_name(pci_dev).

Always refer to the PCI devices by a pointer to the pci_dev structure. All PCI layer functions use this identification and it’s the only reasonable one. Don’t use bus/slot/function numbers except for very special purposes — on systems with multiple primary buses their semantics can be pretty complex.

Don’t try to turn on Fast Back to Back writes in your driver. All devices on the bus need to be capable of doing it, so this is something which needs to be handled by platform and generic code, not individual drivers.

1.9. Vendor and device identifications¶

Do not add new device or vendor IDs to include/linux/pci_ids.h unless they are shared across multiple drivers. You can add private definitions in your driver if they’re helpful, or just use plain hex constants.

The device IDs are arbitrary hex numbers (vendor controlled) and normally used only in a single location, the pci_device_id table.

Please DO submit new vendor/device IDs to http://pci-ids.ucw.cz/. There are mirrors of the pci.ids file at http://pciids.sourceforge.net/ and https://github.com/pciutils/pciids.

1.10. Obsolete functions¶

There are several functions which you might come across when trying to port an old driver to the new PCI interface. They are no longer present in the kernel as they aren’t compatible with hotplug or PCI domains or having sane locking.

pci_find_device()	Superseded by pci_get_device()
pci_find_subsys()	Superseded by pci_get_subsys()
pci_find_slot()	Superseded by pci_get_domain_bus_and_slot()
pci_get_slot()	Superseded by pci_get_domain_bus_and_slot()

The alternative is the traditional PCI device driver that walks PCI device lists. This is still possible but discouraged.

1.11. MMIO Space and «Write Posting»В¶

Converting a driver from using I/O Port space to using MMIO space often requires some additional changes. Specifically, «write posting» needs to be handled. Many drivers (e.g. tg3, acenic, sym53c8xx_2) already do this. I/O Port space guarantees write transactions reach the PCI device before the CPU can continue. Writes to MMIO space allow the CPU to continue before the transaction reaches the PCI device. HW weenies call this «Write Posting» because the write completion is «posted» to the CPU before the transaction has reached its destination.

Thus, timing sensitive code should add readl() where the CPU is expected to wait before doing other work. The classic «bit banging» sequence works fine for I/O Port space:

The same sequence for MMIO space should be:

It is important that «safe_mmio_reg» not have any side effects that interferes with the correct operation of the device.

Another case to watch out for is when resetting a PCI device. Use PCI Configuration space reads to flush the writel(). This will gracefully handle the PCI master abort on all platforms if the PCI device is expected to not respond to a readl(). Most x86 platforms will allow MMIO reads to master abort (a.k.a. «Soft Fail») and return garbage (e.g.

0). But many RISC platforms will crash (a.k.a.»Hard Fail»).

© Copyright The kernel development community.

Источник