On Mon, Oct 04, 2021 at 08:00:03PM -0300, Jason Gunthorpe wrote: > On Sun, Sep 26, 2021 at 11:31:43AM +0300, Leon Romanovsky wrote: > > From: Aharon Landau <aharonl@xxxxxxxxxx> > > > > Currently, if a cache entry is empty, the driver will try to take MRs > > from larger cache entries. This behavior consumes a lot of memory. > > In addition, when searching for an mkey in an entry, the entry is locked. > > When using a multithreaded application with the old behavior, the threads > > will block each other more often, which can hurt performance as can be > > seen in the table below. > > > > Therefore, avoid it by creating a new mkey when the requested cache entry > > is empty. > > > > The test was performed on a machine with > > Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz 44 cores. > > > > Here are the time measures for allocating MRs of 2^6 pages. The search in > > the cache started from entry 6. > > > > +------------+---------------------+---------------------+ > > | | Old behavior | New behavior | > > | +----------+----------+----------+----------+ > > | | 1 thread | 5 thread | 1 thread | 5 thread | > > +============+==========+==========+==========+==========+ > > | 1,000 MRs | 14 ms | 30 ms | 14 ms | 80 ms | > > +------------+----------+----------+----------+----------+ > > | 10,000 MRs | 135 ms | 6 sec | 173 ms | 880 ms | > > +------------+----------+----------+----------+----------+ > > |100,000 MRs | 11.2 sec | 57 sec | 1.74 sec | 8.8 sec | > > +------------+----------+----------+----------+----------+ > > > > Signed-off-by: Aharon Landau <aharonl@xxxxxxxxxx> > > Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxx> > > --- > > drivers/infiniband/hw/mlx5/mr.c | 26 +++++++++----------------- > > 1 file changed, 9 insertions(+), 17 deletions(-) > > I'm surprised the cost is so high, I assume this has alot to do with > repeated calls to queue_adjust_cache_locked()? Maybe this should be > further investigated? I don't think so, most of the overhead comes from entry lock, which effectively stops any change to that shared entry. > > Anyhow, applied to for-next, thanks Thanks > > Jason
