> Manage and allocate pool of objects with given limit on number of
> elements. Gets parameters from rxe_type_info. Pool elements are
> allocated out of a slab cache. Objects that are using this facility
> are: PD, QP, SRQ, CQ, MR, FMR, MW, etc.
>
> Signed-off-by: Kamal Heib <kamalh@xxxxxxxxxxxx>
> Signed-off-by: Amir Vadai <amirv@xxxxxxxxxxxx>
> Signed-off-by: Moni Shoua <monis@xxxxxxxxxxxx>
> Reviewed-by: Haggai Eran <haggaie@xxxxxxxxxxxx>
> ---
> drivers/infiniband/hw/rxe/rxe_pool.c | 510
> +++++++++++++++++++++++++++++++++++
> drivers/infiniband/hw/rxe/rxe_pool.h | 164 +++++++++++
> 2 files changed, 674 insertions(+)
> create mode 100644 drivers/infiniband/hw/rxe/rxe_pool.c
> create mode 100644 drivers/infiniband/hw/rxe/rxe_pool.h
>
> diff --git a/drivers/infiniband/hw/rxe/rxe_pool.c
> b/drivers/infiniband/hw/rxe/rxe_pool.c
> new file mode 100644
> index 0000000..5a7da6b
> --- /dev/null
> +++ b/drivers/infiniband/hw/rxe/rxe_pool.c
> @@ -0,0 +1,510 @@
> +/*
> + * Copyright (c) 2016 Mellanox Technologies Ltd. All rights reserved.
> + * Copyright (c) 2015 System Fabric Works, Inc. All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses. You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * - Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer.
> + *
> + * - Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
> HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + */
> +
> +#include "rxe.h"
> +#include "rxe_loc.h"
> +
> +/* info about object pools
> + * note that mr, fmr and mw share a single index space
> + * so that one can map an lkey to the correct type of object
> + */
> +struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = {
> + [RXE_TYPE_UC] = {
> + .name = "uc",
> + .size = sizeof(struct rxe_ucontext),
> + },
> + [RXE_TYPE_PD] = {
> + .name = "pd",
> + .size = sizeof(struct rxe_pd),
> + },
> + [RXE_TYPE_AH] = {
> + .name = "ah",
> + .size = sizeof(struct rxe_ah),
> + .flags = RXE_POOL_ATOMIC,
> + },
> + [RXE_TYPE_SRQ] = {
> + .name = "srq",
> + .size = sizeof(struct rxe_srq),
> + .flags = RXE_POOL_INDEX,
> + .min_index = RXE_MIN_SRQ_INDEX,
> + .max_index = RXE_MAX_SRQ_INDEX,
> + },
> + [RXE_TYPE_QP] = {
> + .name = "qp",
> + .size = sizeof(struct rxe_qp),
> + .cleanup = rxe_qp_cleanup,
> + .flags = RXE_POOL_INDEX,
> + .min_index = RXE_MIN_QP_INDEX,
> + .max_index = RXE_MAX_QP_INDEX,
> + },
> + [RXE_TYPE_CQ] = {
> + .name = "cq",
> + .size = sizeof(struct rxe_cq),
> + .cleanup = rxe_cq_cleanup,
> + },
> + [RXE_TYPE_MR] = {
> + .name = "mr",
> + .size = sizeof(struct rxe_mem),
> + .cleanup = rxe_mem_cleanup,
> + .flags = RXE_POOL_INDEX,
> + .max_index = RXE_MAX_MR_INDEX,
> + .min_index = RXE_MIN_MR_INDEX,
> + },
> + [RXE_TYPE_FMR] = {
> + .name = "fmr",
> + .size = sizeof(struct rxe_mem),
> + .cleanup = rxe_mem_cleanup,
> + .flags = RXE_POOL_INDEX,
> + .max_index = RXE_MAX_FMR_INDEX,
> + .min_index = RXE_MIN_FMR_INDEX,
> + },
> + [RXE_TYPE_MW] = {
> + .name = "mw",
> + .size = sizeof(struct rxe_mem),
> + .flags = RXE_POOL_INDEX,
> + .max_index = RXE_MAX_MW_INDEX,
> + .min_index = RXE_MIN_MW_INDEX,
> + },
> + [RXE_TYPE_MC_GRP] = {
> + .name = "mc_grp",
> + .size = sizeof(struct rxe_mc_grp),
> + .cleanup = rxe_mc_cleanup,
> + .flags = RXE_POOL_KEY,
> + .key_offset = offsetof(struct rxe_mc_grp, mgid),
> + .key_size = sizeof(union ib_gid),
> + },
> + [RXE_TYPE_MC_ELEM] = {
> + .name = "mc_elem",
> + .size = sizeof(struct rxe_mc_elem),
> + .flags = RXE_POOL_ATOMIC,
> + },
> +};
> +
Perhaps the above slab names should be prefixed with "rxe-"? EG: "rxe-cq",
"rxe-qp", etc. They look very non-descriptive when looking at them in slabinfo:
[root@stevo2 linux-2.6]# head /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>
: tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs>
<num_slabs> <sharedavail>
uc 0 0 352 11 1 : tunables 54 27 8 :
slabdata 0 0 0
cq 6 34 240 17 1 : tunables 120 60 8 :
slabdata 2 2 0
qp 4 6 1920 2 1 : tunables 24 12 8 :
slabdata 3 3 0
pd 4 36 112 36 1 : tunables 120 60 8 :
slabdata 1 1 0
Also when using rxe with nvmf (among other bugs I'm chasing down), when you
unload ib_rxe, there are allocated cq objects left in the the slab cache:
kmem_cache_destroy cq: Slab cache still has objects
CPU: 5 PID: 4147 Comm: rmmod Tainted: G E 4.7.0-rc2-nvmf-all.3+rxe+
#51
Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a 07/09/2015
0000000000000000 ffff881007b33d98 ffffffff812d1359 ffff88103bd65098
ffffea00378983d0 ffff881007b33da8 ffff881078b32240 ffff881007b33df8
ffffffff8117619c ffff881007b33da8 ffff881007b33da8 ffff88103bec6960
Call Trace:
[<ffffffff812d1359>] dump_stack+0x51/0x78
[<ffffffff8117619c>] kmem_cache_destroy+0x12c/0x150
[<ffffffffa04c62cc>] rxe_cache_exit+0x1c/0x40 [ib_rxe]
[<ffffffffa04ce4cf>] rxe_module_exit+0x13/0x23 [ib_rxe]
[<ffffffff810e1d15>] SyS_delete_module+0x185/0x1d0
[<ffffffff8100278e>] ? syscall_trace_enter_phase2+0x6e/0x190
[<ffffffff81002915>] ? syscall_trace_enter+0x65/0x70
[<ffffffff81002d4d>] do_syscall_64+0x6d/0x160
[<ffffffff8161857c>] entry_SYSCALL64_slow_path+0x25/0x25
rxe: unloaded
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
