From: Edward Srouji <edwards@xxxxxxxxxxxx> Add a documentation for shared PD and verbs import usage in Pyverbs. This includes a code snippet to demonstrate a usage example. Reviewed-by: Ido Kalir <idok@xxxxxxxxxxxx> Signed-off-by: Edward Srouji <edwards@xxxxxxxxxxxx> --- Documentation/pyverbs.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md index 77a5406..2516b10 100755 --- a/Documentation/pyverbs.md +++ b/Documentation/pyverbs.md @@ -626,3 +626,43 @@ ctx = Context(name='rocep0s8f0') uar = Mlx5UAR(ctx) uar.close() ``` + +##### Import device, PD and MR +Importing a device, PD and MR enables processes to share their context and then +share PDs and MRs that is associated with. +A process creates a device and then uses some of the Linux systems calls to dup +its 'cmd_fd' member which lets other process to obtain ownership. +Once other process obtains the 'cmd_fd' it can import the device, then PD(s) and +MR(s) to share these objects. +Like in C, Pyverbs users are responsible for unimporting the imported objects +(which will also close the Pyverbs instance in our case) after they finish using +them, and they have to sync between the different processes in order to +coordinate the closure of the objects. +Unlike in C, closing the underlying objects is currently supported only via the +"original" object (meaning only by the process that creates them) and not via +the imported object. This limitation is made because currently there's no +reference or relation between different Pyverbs objects in different processes. +But it's doable and might be added in the future. +Here is a demonstration of importing a device, PD and MR in one process. +```python +from pyverbs.device import Context +from pyverbs.pd import PD +from pyverbs.mr import MR +import pyverbs.enums as e +import os + +ctx = Context(name='ibp0s8f0') +pd = PD(ctx) +mr = MR(pd, 100, e.IBV_ACCESS_LOCAL_WRITE) +cmd_fd_dup = os.dup(ctx.cmd_fd) +improted_ctx = Context(cmd_fd=cmd_fd_dup) +imported_pd = PD(improted_ctx, handle=pd.handle) +imported_mr = MR(imported_pd, handle=mr.handle) +# MRs can be created as usual on the imported PD +secondary_mr = MR(imported_pd, 100, e.IBV_ACCESS_REMOTE_READ) +# Must manually unimport the imported objects (which close the object and frees +# other resources that use them) before closing the "original" objects. +# This prevents unexpected behaviours caused by the GC. +imported_mr.unimport() +imported_pd.unimport() +``` -- 1.8.3.1
