Hi, I am trying to understand if I should expect lower bandwidth from a setup when using IBV_QPT_RAW_PACKET and a steering rule which looks at the MAC addresses of the source and destination NICs. I am trying out an example program from the Mellanox community website which uses these features. For the code example, please see https://community.mellanox.com/s/article/raw-ethernet-programming--basic-introduction---code-example In my test setup, I am using two Mellanox MCX515A-CCAT NICs, which have a maximum bandwidth of 100 Gbps. They are installed in two linux computers which are connected by a single cable (no switch or router). Previously, when I was running tests like ib_send_bw or ib_write_bw and using UD, UC, or RC transport mode, I was seeing bandwidths ~90Gbps or higher. With the example in Raw Ethernet Programming: Basic introduction, after adding some code to count packets and measure time, I am seeing bandwidths around 10 Gbps. I have been playing with different parameters such as MTU, packet size, or IBV_SEND_INLINE. I am wondering if the reduction in bandwidth is due to the packet filtering being done by the steering rule? Or should I expect to see similar bandwidths (~90 Gbps) as in my earlier tests and the problem is one of lack of optimization of my setup? Thanks for any help you can provide.
