Hi Mark, Thanks for the response and the example. Terry On Tue, Mar 17, 2020 at 7:40 PM Mark Bloch <markb@xxxxxxxxxxxx> wrote: > > Hey Terry, > > On 3/17/20 2:07 PM, Terry Toole wrote: > > Hi, > > I am trying to understand if I should expect lower bandwidth from a > > setup when using IBV_QPT_RAW_PACKET and a steering rule which looks at > > the MAC addresses of > > the source and destination NICs. I am trying out an example program > > from the Mellanox community website which uses these features. For the > > code example, please see > > > > https://community.mellanox.com/s/article/raw-ethernet-programming--basic-introduction---code-example > > > > In my test setup, I am using two Mellanox MCX515A-CCAT NICs, which have a > > maximum bandwidth of 100 Gbps. They are installed in two linux computers which > > are connected by a single cable (no switch or router). Previously, > > when I was running > > tests like ib_send_bw or ib_write_bw and using UD, UC, or RC transport > > mode, I was seeing bandwidths ~90Gbps or higher. With the example in > > Raw Ethernet Programming: Basic introduction, after adding some code > > to count packets and measure time, I am seeing bandwidths around 10 > > Gbps. I have been playing with different parameters such as MTU, > > packet size, or IBV_SEND_INLINE. I am wondering if the reduction in > > bandwidth is due to the packet filtering being done by the steering > > While steering requires more work from the HW (if a packet hits too many > steering rules before being directed to a TIR/RQ it might affect the BW) > A single steering rule shouldn't. > > > rule? Or should I expect to see similar bandwidths (~90 Gbps) as in my > > earlier tests and the problem is one of lack of optimization of my > > setup? > > More like lack of optimizations in the test program you've used. > When talking about sending traffic using RAW_ETHERNET QP there are lot of optimizations > that can/should be done. > > If you would like to have a look at a highly optimized datapath from userspace: > https://github.com/DPDK/dpdk/blob/master/drivers/net/mlx5/mlx5_rxtx.c > > With the right code you should have no issue reaching line rate speeds with raw_ethernet QPs > > Mark > > > > > Thanks for any help you can provide. > >
