> From: Bernard Metzler <BMT@xxxxxxxxxxxxxx>
> Sent: Tuesday, September 3, 2019 2:37 PM
>
> External Email
>
> ----------------------------------------------------------------------
> -----"Michal Kalderon" <michal.kalderon@xxxxxxxxxxx> wrote: -----
>
> >To: <mkalderon@xxxxxxxxxxx>, <aelior@xxxxxxxxxxx>, <jgg@xxxxxxxx>,
> ><dledford@xxxxxxxxxx>, <bmt@xxxxxxxxxxxxxx>,
> <galpress@xxxxxxxxxx>,
> ><sleybo@xxxxxxxxxx>, <leon@xxxxxxxxxx>
> >From: "Michal Kalderon" <michal.kalderon@xxxxxxxxxxx>
> >Date: 09/02/2019 06:25PM
> >Cc: <linux-rdma@xxxxxxxxxxxxxxx>, "Michal Kalderon"
> ><michal.kalderon@xxxxxxxxxxx>
> >Subject: [EXTERNAL] [PATCH v9 rdma-next 0/7] RDMA/qedr: Use the
> >doorbell overflow recovery mechanism for RDMA
> >
> >This patch series uses the doorbell overflow recovery mechanism
> >introduced in commit 36907cd5cd72 ("qed: Add doorbell overflow recovery
> >mechanism") for rdma ( RoCE and iWARP )
> >
>
>
> Hi Michal,
>
> I wanted to try out things -- can you please help me:
> Where would that patch apply to? I tried rdma-next master and for-next. I
> am getting conflicts in drivers/infiniband/core/ib_core_uverbs.c. Is there any
> previous patch needed for this series?
It applies to rdma for-next branch. The ib_core_uverbs.c is a new file,
Perhaps you have it from previous apply ?
Please make sure you're synced to head and that you don't have the file.
Let me know if it worked out,
Thanks,
Michal
>
> Thanks very much and best regards,
> Bernard.
>
>
> >The first five patches modify the core code to contain helper functions
> >for managing mmap_xa inserting, getting and freeing entries. The code
> >was based on the code from efa driver.
> >There is still an open discussion on whether we should take this even
> >further and make the entire mmap generic. Until a decision is made, I
> >only created the database API and modified the efa, qedr, siw driver to
> >use it. The functions are integrated with the umap mechanism.
> >
> >The doorbell recovery code is based on the common code.
> >
> >Efa driver was compile tested and checked only modprobe/rmmod.
> >SIW was compile tested and checked only modprobe/rmmod.
> >
> >rdma-core pull request #493
> >
> >Changes from V8:
> >- CORE changes
> > - Fix race between getting an entry and deleting it. Increase
> > the refcount under the lock only if it is not zero. Erase all
> >entries
> > with __xa_erase instead of xa_erase and take the lock outside the
> >loop.
> > - Fix comment when erasing all the xa_entries of a single mmap_entry.
> > - Take comment out of loop
> > - Change length field in driver structures to be size_t instead of
> >u64
> > suggested by Bernard Metzler
> > - Change do..while(true) to while(true)
> >- COMMON driver changes
> > - Change mmap length to be size_t instead of u64.
> > - In mmap, call put_entry if there is a length error.
> >- EFA changes:
> > - Reverse mmap entries remove order.
> > - Give meaningful label names in create_qp error flows.
> > - In error flow undo change that frees pages based only on key and
> > make sure rq_size > 0 first.
> > - Fix xmas tree alignment, move ucontext initialization to
> >declaration
> > line.
> >- SIW changes:
> > - Changes received from Bernard Metzler
> > - make the siw_user_mmap_entry.address a void *, which
> > naturally fits with remap_vmalloc_range. also avoids
> > other casting during resource address assignment.
> > - do not kfree SQ/RQ/CQ/SRQ in preparation of mmap.
> > Those resources are always further needed ;)
> > - Fix check for correct mmap range in siw_mmap().
> > - entry->length is the object length. We have to
> > expand to PAGE_ALIGN(entry->length), since mmap
> > comes with complete page(s) containing the
> > object.
> > - put mmap_entry if that check fails. Otherwise
> > entry object ref counting screws up, and later
> > crashes during context close.
> > - simplify siw_mmap_free() - it must just free
> > the entry.
> > - Change length to size_t instead of u64
> >
> >Changes from V7:
> >- Remove license text, SPDX id should suffice.
> >- Fix some comments text.
> >- Add comment regarding vm_ops being set in ib_uverbs_mmap.
> >- Allocate the rdma_user_mmap_entry in the driver and not in the
> > ib_core_uverbs. This lead to defining three new structures per driver
> > and seperating the fields between the driver private structures and
> > the common rdma_user_mmap_entry. Freeing the entry was also moved
> > to the drivers.
> >- Fix bug found by Gal Pressman. Call mmap_free only once per entry.
> >- Add a mutex around xa_mmap insert to assure threads won't intefere
> > while the xa lock is released when inserting an entry into the range.
> >- Modify the insert algorithm to be more elegant using the
> > xas_next_entry instead of foreach.
> >- Remove the rdma_user_mmap_entries_remove_free function, now that
> >umap.
> > and mmap_xa are integrated we should not have any entries in the
> >mmap_xa
> > when ucontext is released. Replace the function with a
> >WARN_ON(!xa_empty).
> >- Rdma_umap_open needs to reset the vm_private_data before initializing
> >it.
> >- Decrease rdma_user_mmap_entry reference count on mmap
> disassociate.
> >- Remove WARN_ON(!kref_read) this is checked when kref debug is on.
> >- Remove some redundant defines from ib_verbs.h.
> >- Better error handling for efa create qp flow.
> >- Add a function that wraps the entry allocation and
> >rdma_user_mmap_entry_insert
> > which is used in all places that need to add an entry to the xarray.
> >- Remove rq_entry_inserted field in efa create qp flow.
> >- Add mmap_free to siw and free the memory only on mmap free and not
> >before.
> >
> >Changes from V6:
> >- Modified series description to be closer to what the series is now.
> >- Create a new file for the new rdma_user_mmap function. The file
> > is called ib_uverbs_core. This file should contain functions related
> > to user which are called by hw to eventually enable ib_uverbs to be
> > optional.
> >- Modify SIW driver to use new mmap api.
> >- When calculating number of pages, need to round it up to PAGE_SIZE.
> >- Integrate the mmap_xa and umap mechanism so that the entries in
> > mmap_xa now have a reference count and can be removed. Previously
> > entries existed until context was destroyed. This modified the
> > algorithm for allocating a free page range.
> >- Modify algorithm for inserting an entry into the mmap_xa.
> >- Rdma_umap_priv is now also used for all mmaps done using the
> > mmap_xa helpers.
> >- Move remove_free header to core_priv.
> >- Rdma_user_mmap_entry now has a kref that is increase on mmap
> > and umap_open and decreased on umap_close.
> >- Modify efa + qedr to remove the entry from xa_map. This will
> > decrease the refcnt and free memory only if refcnt is zero.
> >- Rdma_user_mmap_io slightly modified to enable drivers not using
> > the xa_mmap API to continue using it.
> >- Modify page allocation for user to use GFP_USER instead of GFP_KERNEL
> >
> >Changes from V5:
> >- Switch between driver dealloc_ucontext and mmap_entries_remove call.
> >- No need to verify the key after using the key to load an entry from
> > the mmap_xa.
> >- Change mmap_free api to pass an 'entry' object.
> >- Add documentation for mmap_free and for newly exported functions.
> >- Fix some extra/missing line breaks.
> >
> >Changes from V4:
> >- Add common mmap database and cookie helper functions.
> >
> >Changes from V3:
> >- Remove casts from void to u8. Pointer arithmetic can be done on void
> >- rebase to tip of rdma-next
> >
> >Changes from V2:
> >- Don't use long-lived kmap. Instead use user-trigger mmap for the
> > doorbell recovery entries.
> >- Modify dpi_addr to be denoted with __iomem and avoid redundant
> > casts
> >
> >Changes from V1:
> >- call kmap to map virtual address into kernel space
> >- modify db_rec_delete to be void
> >- remove some cpu_to_le16 that were added to previous patch which are
> > correct but not related to the overflow recovery mechanism. Will be
> > submitted as part of a different patch
> >
> >
> >Michal Kalderon (7):
> > RDMA/core: Move core content from ib_uverbs to ib_core
> > RDMA/core: Create mmap database and cookie helper functions
> > RDMA/efa: Use the common mmap_xa helpers
> > RDMA/siw: Use the common mmap_xa helpers
> > RDMA/qedr: Use the common mmap API
> > RDMA/qedr: Add doorbell overflow recovery support
> > RDMA/qedr: Add iWARP doorbell recovery support
> >
> > drivers/infiniband/core/Makefile | 2 +-
> > drivers/infiniband/core/core_priv.h | 16 +
> > drivers/infiniband/core/device.c | 1 +
> > drivers/infiniband/core/ib_core_uverbs.c | 353 ++++++++++++++++++++
> > drivers/infiniband/core/rdma_core.c | 1 +
> > drivers/infiniband/core/uverbs_cmd.c | 1 +
> > drivers/infiniband/core/uverbs_main.c | 97 +-----
> > drivers/infiniband/hw/efa/efa.h | 18 +-
> > drivers/infiniband/hw/efa/efa_main.c | 1 +
> > drivers/infiniband/hw/efa/efa_verbs.c | 360 ++++++++++----------
> > drivers/infiniband/hw/qedr/main.c | 1 +
> > drivers/infiniband/hw/qedr/qedr.h | 45 ++-
> > drivers/infiniband/hw/qedr/verbs.c | 544
> >++++++++++++++++++++++---------
> > drivers/infiniband/hw/qedr/verbs.h | 3 +-
> > drivers/infiniband/sw/siw/siw.h | 20 +-
> > drivers/infiniband/sw/siw/siw_main.c | 1 +
> > drivers/infiniband/sw/siw/siw_verbs.c | 216 ++++++------
> > drivers/infiniband/sw/siw/siw_verbs.h | 1 +
> > include/rdma/ib_verbs.h | 37 ++-
> > include/uapi/rdma/qedr-abi.h | 25 ++
> > 20 files changed, 1196 insertions(+), 547 deletions(-) create mode
> > 100644 drivers/infiniband/core/ib_core_uverbs.c
> >
> >--
> >2.14.5
> >
> >
