On 4/6/2021 2:53 PM, Jason Gunthorpe wrote:
On Tue, Apr 06, 2021 at 08:09:43AM +0300, Leon Romanovsky wrote:
On Tue, Apr 06, 2021 at 10:37:38AM +0800, Honggang LI wrote:
On Mon, Apr 05, 2021 at 08:23:54AM +0300, Leon Romanovsky wrote:
From: Leon Romanovsky <leonro@xxxxxxxxxx>
From Avihai,
Relaxed Ordering is a PCIe mechanism that relaxes the strict ordering
imposed on PCI transactions, and thus, can improve performance.
Until now, relaxed ordering could be set only by user space applications
for user MRs. The following patch series enables relaxed ordering for the
kernel ULPs as well. Relaxed ordering is an optional capability, and as
such, it is ignored by vendors that don't support it.
The following test results show the performance improvement achieved
Did you test this patchset with CPU does not support relaxed ordering?
I don't think so, the CPUs that don't support RO are Intel's fourth/fifth-generation
and they are not interesting from performance point of view.
We observed significantly performance degradation when run perftest with
relaxed ordering enabled over old CPU.
https://github.com/linux-rdma/perftest/issues/116
The perftest is slightly different, but you pointed to the valid point.
We forgot to call pcie_relaxed_ordering_enabled() before setting RO bit
and arguably this was needed to be done in perftest too.
No, the PCI device should not have the RO bit set in this situation.
It is something mlx5_core needs to do. We can't push this into
applications.
pcie_relaxed_ordering_enabled is called in
drivers/net/ethernet/mellanox/mlx5/core/en_common.c so probably need to
move it to
mlx5_core in this series.
There should be no performance difference from asking for
IBV_ACCESS_RELAXED_ORDERING when RO is disabled at the PCI config and
not asking for it at all.
Either the platform has working relaxed ordering that gives a
performance gain and the RO config spec bit should be set, or it
doesn't and the bit should be clear.
is this the case today ?
This is not something to decide in userspace, or in RDMA. At worst it
becomes another platform specific PCI tunable people have to set.
I thought the old haswell systems were quirked to disable RO globally
anyhow?
Jason
