-----"Michal Kalderon" <michal.kalderon@xxxxxxxxxxx> wrote: -----
>To: <mkalderon@xxxxxxxxxxx>, <aelior@xxxxxxxxxxx>, <jgg@xxxxxxxx>,
><dledford@xxxxxxxxxx>, <bmt@xxxxxxxxxxxxxx>, <galpress@xxxxxxxxxx>,
><sleybo@xxxxxxxxxx>, <leon@xxxxxxxxxx>
>From: "Michal Kalderon" <michal.kalderon@xxxxxxxxxxx>
>Date: 09/02/2019 06:25PM
>Cc: <linux-rdma@xxxxxxxxxxxxxxx>, "Michal Kalderon"
><michal.kalderon@xxxxxxxxxxx>
>Subject: [EXTERNAL] [PATCH v9 rdma-next 0/7] RDMA/qedr: Use the
>doorbell overflow recovery mechanism for RDMA
>
>This patch series uses the doorbell overflow recovery mechanism
>introduced in
>commit 36907cd5cd72 ("qed: Add doorbell overflow recovery mechanism")
>for rdma ( RoCE and iWARP )
>
Hi Michal,
I wanted to try out things -- can you please help me:
Where would that patch apply to? I tried rdma-next
master and for-next. I am getting conflicts in
drivers/infiniband/core/ib_core_uverbs.c. Is there
any previous patch needed for this series?
Thanks very much and best regards,
Bernard.
>The first five patches modify the core code to contain helper
>functions for managing mmap_xa inserting, getting and freeing
>entries. The code was based on the code from efa driver.
>There is still an open discussion on whether we should take
>this even further and make the entire mmap generic. Until a
>decision is made, I only created the database API and modified
>the efa, qedr, siw driver to use it. The functions are integrated
>with the umap mechanism.
>
>The doorbell recovery code is based on the common code.
>
>Efa driver was compile tested and checked only modprobe/rmmod.
>SIW was compile tested and checked only modprobe/rmmod.
>
>rdma-core pull request #493
>
>Changes from V8:
>- CORE changes
> - Fix race between getting an entry and deleting it. Increase
> the refcount under the lock only if it is not zero. Erase all
>entries
> with __xa_erase instead of xa_erase and take the lock outside the
>loop.
> - Fix comment when erasing all the xa_entries of a single
>mmap_entry.
> - Take comment out of loop
> - Change length field in driver structures to be size_t instead of
>u64
> suggested by Bernard Metzler
> - Change do..while(true) to while(true)
>- COMMON driver changes
> - Change mmap length to be size_t instead of u64.
> - In mmap, call put_entry if there is a length error.
>- EFA changes:
> - Reverse mmap entries remove order.
> - Give meaningful label names in create_qp error flows.
> - In error flow undo change that frees pages based only on key and
> make sure rq_size > 0 first.
> - Fix xmas tree alignment, move ucontext initialization to
>declaration
> line.
>- SIW changes:
> - Changes received from Bernard Metzler
> - make the siw_user_mmap_entry.address a void *, which
> naturally fits with remap_vmalloc_range. also avoids
> other casting during resource address assignment.
> - do not kfree SQ/RQ/CQ/SRQ in preparation of mmap.
> Those resources are always further needed ;)
> - Fix check for correct mmap range in siw_mmap().
> - entry->length is the object length. We have to
> expand to PAGE_ALIGN(entry->length), since mmap
> comes with complete page(s) containing the
> object.
> - put mmap_entry if that check fails. Otherwise
> entry object ref counting screws up, and later
> crashes during context close.
> - simplify siw_mmap_free() - it must just free
> the entry.
> - Change length to size_t instead of u64
>
>Changes from V7:
>- Remove license text, SPDX id should suffice.
>- Fix some comments text.
>- Add comment regarding vm_ops being set in ib_uverbs_mmap.
>- Allocate the rdma_user_mmap_entry in the driver and not in the
> ib_core_uverbs. This lead to defining three new structures per
>driver
> and seperating the fields between the driver private structures and
> the common rdma_user_mmap_entry. Freeing the entry was also moved
> to the drivers.
>- Fix bug found by Gal Pressman. Call mmap_free only once per entry.
>- Add a mutex around xa_mmap insert to assure threads won't intefere
> while the xa lock is released when inserting an entry into the
>range.
>- Modify the insert algorithm to be more elegant using the
> xas_next_entry instead of foreach.
>- Remove the rdma_user_mmap_entries_remove_free function, now that
>umap.
> and mmap_xa are integrated we should not have any entries in the
>mmap_xa
> when ucontext is released. Replace the function with a
>WARN_ON(!xa_empty).
>- Rdma_umap_open needs to reset the vm_private_data before
>initializing it.
>- Decrease rdma_user_mmap_entry reference count on mmap disassociate.
>- Remove WARN_ON(!kref_read) this is checked when kref debug is on.
>- Remove some redundant defines from ib_verbs.h.
>- Better error handling for efa create qp flow.
>- Add a function that wraps the entry allocation and
>rdma_user_mmap_entry_insert
> which is used in all places that need to add an entry to the
>xarray.
>- Remove rq_entry_inserted field in efa create qp flow.
>- Add mmap_free to siw and free the memory only on mmap free and not
>before.
>
>Changes from V6:
>- Modified series description to be closer to what the series is now.
>- Create a new file for the new rdma_user_mmap function. The file
> is called ib_uverbs_core. This file should contain functions
>related
> to user which are called by hw to eventually enable ib_uverbs to be
> optional.
>- Modify SIW driver to use new mmap api.
>- When calculating number of pages, need to round it up to PAGE_SIZE.
>- Integrate the mmap_xa and umap mechanism so that the entries in
> mmap_xa now have a reference count and can be removed. Previously
> entries existed until context was destroyed. This modified the
> algorithm for allocating a free page range.
>- Modify algorithm for inserting an entry into the mmap_xa.
>- Rdma_umap_priv is now also used for all mmaps done using the
> mmap_xa helpers.
>- Move remove_free header to core_priv.
>- Rdma_user_mmap_entry now has a kref that is increase on mmap
> and umap_open and decreased on umap_close.
>- Modify efa + qedr to remove the entry from xa_map. This will
> decrease the refcnt and free memory only if refcnt is zero.
>- Rdma_user_mmap_io slightly modified to enable drivers not using
> the xa_mmap API to continue using it.
>- Modify page allocation for user to use GFP_USER instead of
>GFP_KERNEL
>
>Changes from V5:
>- Switch between driver dealloc_ucontext and mmap_entries_remove
>call.
>- No need to verify the key after using the key to load an entry from
> the mmap_xa.
>- Change mmap_free api to pass an 'entry' object.
>- Add documentation for mmap_free and for newly exported functions.
>- Fix some extra/missing line breaks.
>
>Changes from V4:
>- Add common mmap database and cookie helper functions.
>
>Changes from V3:
>- Remove casts from void to u8. Pointer arithmetic can be done on
>void
>- rebase to tip of rdma-next
>
>Changes from V2:
>- Don't use long-lived kmap. Instead use user-trigger mmap for the
> doorbell recovery entries.
>- Modify dpi_addr to be denoted with __iomem and avoid redundant
> casts
>
>Changes from V1:
>- call kmap to map virtual address into kernel space
>- modify db_rec_delete to be void
>- remove some cpu_to_le16 that were added to previous patch which are
> correct but not related to the overflow recovery mechanism. Will be
> submitted as part of a different patch
>
>
>Michal Kalderon (7):
> RDMA/core: Move core content from ib_uverbs to ib_core
> RDMA/core: Create mmap database and cookie helper functions
> RDMA/efa: Use the common mmap_xa helpers
> RDMA/siw: Use the common mmap_xa helpers
> RDMA/qedr: Use the common mmap API
> RDMA/qedr: Add doorbell overflow recovery support
> RDMA/qedr: Add iWARP doorbell recovery support
>
> drivers/infiniband/core/Makefile | 2 +-
> drivers/infiniband/core/core_priv.h | 16 +
> drivers/infiniband/core/device.c | 1 +
> drivers/infiniband/core/ib_core_uverbs.c | 353 ++++++++++++++++++++
> drivers/infiniband/core/rdma_core.c | 1 +
> drivers/infiniband/core/uverbs_cmd.c | 1 +
> drivers/infiniband/core/uverbs_main.c | 97 +-----
> drivers/infiniband/hw/efa/efa.h | 18 +-
> drivers/infiniband/hw/efa/efa_main.c | 1 +
> drivers/infiniband/hw/efa/efa_verbs.c | 360 ++++++++++----------
> drivers/infiniband/hw/qedr/main.c | 1 +
> drivers/infiniband/hw/qedr/qedr.h | 45 ++-
> drivers/infiniband/hw/qedr/verbs.c | 544
>++++++++++++++++++++++---------
> drivers/infiniband/hw/qedr/verbs.h | 3 +-
> drivers/infiniband/sw/siw/siw.h | 20 +-
> drivers/infiniband/sw/siw/siw_main.c | 1 +
> drivers/infiniband/sw/siw/siw_verbs.c | 216 ++++++------
> drivers/infiniband/sw/siw/siw_verbs.h | 1 +
> include/rdma/ib_verbs.h | 37 ++-
> include/uapi/rdma/qedr-abi.h | 25 ++
> 20 files changed, 1196 insertions(+), 547 deletions(-)
> create mode 100644 drivers/infiniband/core/ib_core_uverbs.c
>
>--
>2.14.5
>
>
