> hfi1 calls to the kernel for data path operations - that is "fake" in my book. Verbs was always about avoiding that kernel transition, to put it back in betrays the spirit. > So a kernel call for rv, or the hfi cdev, or the verbs post-send is really all a wash. To be clear, different vendors have different priorities and hence different HW designs and approaches. hfi1 approached the HPC latency needs with a uniquely scalable approach with very low latency @scale. Ironically, other vendors have since replicated some of those mechanisms with their own proprietary mechanisms, such as UD-X. hfi1 approached storage needs and large message HPC transfers with a direct data placement mechanism (aka RDMA). It fully supported the verbs API and met the performance needs of it's customers with attractive price and power for its target markets. Kaike sited hfi1 as just one example of a RDMA verbs device which rv can function for. Various network and server vendors will each make their own recommendations for software configs and various end users will each have their own preferences and requirements. Todd
