> -----Original Message----- > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > Sent: Tuesday, August 27, 2024 4:52 PM > To: acpica-devel@xxxxxxxxxxxxxxx; Guohanjun (Hanjun Guo) > <guohanjun@xxxxxxxxxx>; iommu@xxxxxxxxxxxxxxx; Joerg Roedel > <joro@xxxxxxxxxx>; Kevin Tian <kevin.tian@xxxxxxxxx>; > kvm@xxxxxxxxxxxxxxx; Len Brown <lenb@xxxxxxxxxx>; linux- > acpi@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; Lorenzo Pieralisi > <lpieralisi@xxxxxxxxxx>; Rafael J. Wysocki <rafael@xxxxxxxxxx>; Robert > Moore <robert.moore@xxxxxxxxx>; Robin Murphy > <robin.murphy@xxxxxxx>; Sudeep Holla <sudeep.holla@xxxxxxx>; Will > Deacon <will@xxxxxxxxxx> > Cc: Alex Williamson <alex.williamson@xxxxxxxxxx>; Eric Auger > <eric.auger@xxxxxxxxxx>; Jean-Philippe Brucker <jean- > philippe@xxxxxxxxxx>; Moritz Fischer <mdf@xxxxxxxxxx>; Michael Shavit > <mshavit@xxxxxxxxxx>; Nicolin Chen <nicolinc@xxxxxxxxxx>; > patches@xxxxxxxxxxxxxxx; Shameerali Kolothum Thodi > <shameerali.kolothum.thodi@xxxxxxxxxx>; Mostafa Saleh > <smostafa@xxxxxxxxxx> > Subject: [PATCH v2 2/8] iommu/arm-smmu-v3: Use S2FWB when available > > Force Write Back (FWB) changes how the S2 IOPTE's MemAttr field > works. When S2FWB is supported and enabled the IOPTE will force cachable > access to IOMMU_CACHE memory when nesting with a S1 and deny cachable > access otherwise. > > When using a single stage of translation, a simple S2 domain, it doesn't > change anything as it is just a different encoding for the exsting mapping > of the IOMMU protection flags to cachability attributes. > > However, when used with a nested S1, FWB has the effect of preventing the > guest from choosing a MemAttr in it's S1 that would cause ordinary DMA to > bypass the cache. Consistent with KVM we wish to deny the guest the > ability to become incoherent with cached memory the hypervisor believes is > cachable so we don't have to flush it. > > Turn on S2FWB whenever the SMMU supports it and use it for all S2 > mappings. > > Signed-off-by: Jason Gunthorpe <jgg@xxxxxxxxxx> > --- (...) > @@ -932,7 +948,8 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg > *cfg, void *cookie) > if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | > IO_PGTABLE_QUIRK_ARM_TTBR1 | > IO_PGTABLE_QUIRK_ARM_OUTER_WBWA | > - IO_PGTABLE_QUIRK_ARM_HD)) > + IO_PGTABLE_QUIRK_ARM_HD | > + IO_PGTABLE_QUIRK_ARM_S2FWB)) > return NULL; This should be added to arm_64_lpae_alloc_pgtable_s2(), not here. With the above fixed, I was able to assign a n/w VF dev to a Guest on a test hardware that supports S2FWB. However host kernel has this WARN message: [ 1546.165105] WARNING: CPU: 5 PID: 7047 at drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1086 arm_smmu_entry_qword_diff+0x124/0x138 .... [ 1546.330312] arm_smmu_entry_qword_diff+0x124/0x138 [ 1546.335090] arm_smmu_write_entry+0x38/0x22c [ 1546.339346] arm_smmu_install_ste_for_dev+0x158/0x1ac [ 1546.344383] arm_smmu_attach_dev+0x138/0x240 [ 1546.348639] __iommu_device_set_domain+0x7c/0x11c [ 1546.353330] __iommu_group_set_domain_internal+0x60/0x134 [ 1546.358714] iommu_group_replace_domain+0x3c/0x68 [ 1546.363404] iommufd_device_do_replace+0x334/0x398 [ 1546.368181] iommufd_device_change_pt+0x26c/0x650 [ 1546.372871] iommufd_device_replace+0x18/0x24 [ 1546.377214] vfio_iommufd_physical_attach_ioas+0x28/0x68 [ 1546.382514] vfio_df_ioctl_attach_pt+0x98/0x170 And when I tried to use the assigned n/w dev, it seems to do a reset continuously. root@localhost:/# ping 150.0.124.42 PING 150.0.124.42 (150.0.124.42): 56 data bytes 64 bytes from 150.0.124.42: seq=0 ttl=64 time=47.648 ms [ 1395.958630] hns3 0000:c2:00.0 eth1: NETDEV WATCHDOG: CPU: 1: transmit queue 10 timed out 5260 ms [ 1395.960187] hns3 0000:c2:00.0 eth1: DQL info last_cnt: 42, queued: 42, adj_limit: 0, completed: 0 [ 1395.961758] hns3 0000:c2:00.0 eth1: queue state: 0x6, delta msecs: 5260 [ 1395.962925] hns3 0000:c2:00.0 eth1: tx_timeout count: 1, queue id: 10, SW_NTU: 0x1, SW_NTC: 0x0, napi state: 16 [ 1395.964677] hns3 0000:c2:00.0 eth1: tx_pkts: 0, tx_bytes: 0, sw_err_cnt: 0, tx_pending: 0 [ 1395.966114] hns3 0000:c2:00.0 eth1: seg_pkt_cnt: 0, tx_more: 0, restart_queue: 0, tx_busy: 0 [ 1395.967598] hns3 0000:c2:00.0 eth1: tx_push: 1, tx_mem_doorbell: 0 [ 1395.968687] hns3 0000:c2:00.0 eth1: BD_NUM: 0x7f HW_HEAD: 0x0, HW_TAIL: 0x0, BD_ERR: 0x0, INT: 0x1 [ 1395.970291] hns3 0000:c2:00.0 eth1: RING_EN: 0x1, TC: 0x0, FBD_NUM: 0x0 FBD_OFT: 0x0, EBD_NUM: 0x400, EBD_OFT: 0x0 [ 1395.972134] hns3 0000:c2:00.0: received reset request from VF enet All this works fine on a hardware without S2FWB though. Also on this test hardware, it works fine with legacy VFIO assignment. Not debugged further. Please let me know if you have any hunch. Thanks, Shameer
