On Fri, Mar 19, 2021 at 08:12:18PM +0000, Rimmer, Todd wrote: > > hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. > So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash. > > To be clear, different vendors have different priorities and hence > different HW designs and approaches. hfi1 approached the HPC > latency needs with a uniquely scalable approach with very low > latency @scale. Yes, we all know the marketing spin the vendors like to use here. A kernel transition is a kernel transition, and the one in the HFI verbs path through all the common code is particularly expensive. I'm suprirsed to hear someone advocate that is a good thing when we were all told that the hfi1 cdev *must* exist because the kernel transition through verbs was far to expensive. > Ironically, other vendors have since replicated some > of those mechanisms with their own proprietary mechanisms, such as > UD-X. What is a UD-X? > Kaike sited hfi1 as just one example of a RDMA verbs device which rv > can function for. rv seems to completely destroy alot of the HPC performance offloads that vendors are layering on RC QPs (see what hns and mlx5 are doing), so I have doubt about this. It would be better if you could get at least one other device to agree this is reasonable. Or work with something that already exists for bulk messaging like RDS. Jason
