On Thu, Feb 10, 2022 at 05:38:06PM -0600, Bjorn Helgaas wrote:
> If a device sets the No Command Completed Support bit in the Slot
> Capabilities register (PCIe r6.0, sec 7.5.3.9), it does not generate
> software notifications when a command is completed, and software can
> write all fields of the Slot Control register without any delays.
>
> Since we only need to wait for command completion on devices that do *not*
> set the No Command Completed Support bit, there's no need to even set the
> ctrl->cmd_busy bit that tracks when the device is busy handling a command.
>
> Set the ctrl->cmd_busy bit only when we need to wait for a command to
> complete. This shouldn't make much functional difference, but does avoid
> an smp_mb() for controllers that set PCI_EXP_SLTCAP_NCCS.
[...]
> --- a/drivers/pci/hotplug/pciehp_hpc.c
> +++ b/drivers/pci/hotplug/pciehp_hpc.c
> @@ -114,13 +114,6 @@ static void pcie_wait_cmd(struct controller *ctrl)
> unsigned long now, timeout;
> int rc;
>
> - /*
> - * If the controller does not generate notifications for command
> - * completions, we never need to wait between writes.
> - */
> - if (NO_CMD_CMPL(ctrl))
> - return;
> -
> if (!ctrl->cmd_busy)
> return;
>
> @@ -173,8 +166,17 @@ static void pcie_do_write_cmd(struct controller *ctrl, u16 cmd,
> slot_ctrl_orig = slot_ctrl;
> slot_ctrl &= ~mask;
> slot_ctrl |= (cmd & mask);
> - ctrl->cmd_busy = 1;
> - smp_mb();
> +
> + /*
> + * If controller generates Command Completed events, we must wait
> + * for it to finish one command before sending another, so we need
> + * to keep track of when the controller is busy.
> + */
> + if (!NO_CMD_CMPL(ctrl)) {
> + ctrl->cmd_busy = 1;
> + smp_mb();
> + }
> +
> ctrl->slot_ctrl = slot_ctrl;
> pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, slot_ctrl);
> ctrl->cmd_started = jiffies;
Looks good in principle, however ctrl->cmd_busy is set to 1 in two
other places, namely pciehp_resume_noirq() and pciehp_runtime_resume()
in pciehp_core.c. Those two need to be amended as well.
Note that a few lines after the hunk above, the ctrl->cmd_busy flag
is reverted back to 0 on certain controllers:
if (pdev->broken_cmd_compl &&
(slot_ctrl_orig & CC_ERRATUM_MASK) == (slot_ctrl & CC_ERRATUM_MASK))
ctrl->cmd_busy = 0;
That could be combined with your "if (!NO_CMD_CMPL(ctrl))" check:
Basically, "set cmd_busy to 1 if the controller supports Command
Completed events and it's not affected by this erratum".
Perhaps it makes sense to put that in a pciehp_has_command_completion()
helper (or whatever name you prefer), which could then also be called
from pciehp_resume_noirq() / pciehp_runtime_resume().
Thanks (and sorry for the delay),
Lukas
