On Wed, Jan 4, 2023 at 6:50 AM Lazar, Lijo <lijo.lazar@xxxxxxx> wrote: > > > > On 1/4/2023 4:11 AM, Marek Olšák wrote: > > I see. Well, those sysfs files are not usable, and I don't think it > > would be important even if they were usable, but for completeness: > > > > The ioctl returns: > > pcie_gen = 1 > > pcie_num_lanes = 16 > > > > Theoretical bandwidth from those values: 4.0 GB/s > > My DMA test shows this write bandwidth: 3.5 GB/s > > It matches the expectation. > > > > Let's see the devices (there is only 1 GPU Navi21 in the system): > > $ lspci |egrep '(PCI|VGA).*Navi' > > 0a:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL > > Upstream Port of PCI Express Switch (rev c3) > > 0b:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL > > Downstream Port of PCI Express Switch > > 0c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > > [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c3) > > > > Let's read sysfs: > > > > $ cat /sys/bus/pci/devices/0000:0a:00.0/current_link_width > > 16 > > $ cat /sys/bus/pci/devices/0000:0b:00.0/current_link_width > > 16 > > $ cat /sys/bus/pci/devices/0000:0c:00.0/current_link_width > > 16 > > $ cat /sys/bus/pci/devices/0000:0a:00.0/current_link_speed > > 2.5 GT/s PCIe > > $ cat /sys/bus/pci/devices/0000:0b:00.0/current_link_speed > > 16.0 GT/s PCIe > > $ cat /sys/bus/pci/devices/0000:0c:00.0/current_link_speed > > 16.0 GT/s PCIe > > > > Problem 1: None of the speed numbers match 4 GB/s. > > US bridge = 2.5GT/s means operating at PCIe Gen 1 speed. Total > theoretical bandwidth is then derived based on encoding and total number > of lanes. > > > Problem 2: Userspace doesn't know the bus index of the bridges, and it's > > not clear which bridge should be used. > > In general, modern ones have this arch= US->DS->EP. US is the one > connected to physical link. navi and newer have a US and DS bridge, while vega only have a single bridge. Pre-vega are just exposed as endpoints. The EP to bridge links are always the max supported lanes/gen because they are virtual links. The US is the actual physical link to the system. Also note that the PMFW will change the gen speed on the fly dynamically based on traffic so just reading it at idle will always show the slowest speed. Alex > > > Problem 3: The PCIe gen number is missing. > > Current link speed is based on whether it's Gen1/2/3/4/5. > > BTW, your patch makes use of capabilities flags which gives the maximum > supported speed/width by the device. It may not necessarily reflect the > current speed/width negotiated. I guess in NV, this info is already > obtained from PMFW and made available through metrics table. > > Thanks, > Lijo > > > > > That's all irrelevant because all information should be queryable via > > the INFO ioctl. It doesn't matter what sysfs contains because UMDs > > shouldn't have to open and parse extra files just to read a couple of > > integers. > > > > Marek > > > > > > On Tue, Jan 3, 2023 at 3:31 AM Christian König > > <ckoenig.leichtzumerken@xxxxxxxxx > > <mailto:ckoenig.leichtzumerken@xxxxxxxxx>> wrote: > > > > Sure they can, those files are accessible to everyone. > > > > The massive advantage is that this is standard for all PCIe devices, > > so it should work vendor independent. > > > > Christian. > > > > Am 02.01.23 um 18:55 schrieb Marek Olšák: > >> Userspace drivers can't access sysfs. > >> > >> Marek > >> > >> On Mon, Jan 2, 2023, 10:54 Christian König > >> <ckoenig.leichtzumerken@xxxxxxxxx > >> <mailto:ckoenig.leichtzumerken@xxxxxxxxx>> wrote: > >> > >> That stuff is already available as current_link_speed and > >> current_link_width in sysfs. > >> > >> I'm a bit reluctant duplicating this information in the IOCTL > >> interface. > >> > >> Christian. > >> > >> Am 30.12.22 um 23:07 schrieb Marek Olšák: > >>> For computing PCIe bandwidth in userspace and troubleshooting > >>> PCIe > >>> bandwidth issues. > >>> > >>> For example, my Navi21 has been limited to PCIe gen 1 and this is > >>> the first time I noticed it after 2 years. > >>> > >>> Note that this intentionally fills a hole and padding > >>> in drm_amdgpu_info_device. > >>> > >>> Signed-off-by: Marek Olšák <marek.olsak@xxxxxxx > >>> <mailto:marek.olsak@xxxxxxx>> > >>> > >>> The patch is attached. > >>> > >>> Marek > >>> > >> > >
