On Mon, Oct 10, 2016 at 2:22 PM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: > On Mon, Oct 10, 2016 at 02:05:27PM +0530, Parav Pandit wrote: >> On Mon, Oct 10, 2016 at 1:03 PM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: >> > On Mon, Oct 10, 2016 at 11:59:45AM +0530, Parav Pandit wrote: >> >> Hi Leon, >> >> >> >> On Mon, Oct 10, 2016 at 10:16 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: >> >> > On Thu, Oct 06, 2016 at 07:19:24PM +0530, Parav Pandit wrote: >> >> >> Hi Leon, >> >> >> >> >> >> On Wed, Oct 5, 2016 at 4:52 PM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: >> >> >> > On Wed, Aug 31, 2016 at 02:07:24PM +0530, Parav Pandit wrote: >> >> >> >> rdmacg: IB/core: rdma controller support >> >> >> >> >> >> >> >> Overview: >> >> >> >> Currently user space applications can easily take away all the rdma >> >> >> >> device specific resources such as AH, CQ, QP, MR etc. Due to which other >> >> >> >> applications in other cgroup or kernel space ULPs may not even get chance >> >> >> >> to allocate any rdma resources. This results into service unavailibility. >> >> >> >> >> >> >> >> RDMA cgroup addresses this issue by allowing resource accounting, >> >> >> >> limit enforcement on per cgroup, per rdma device basis. >> >> >> >> >> >> >> >> RDMA uverbs layer will enforce limits on well defined RDMA verb >> >> >> >> resources without any HCA vendor device driver involvement. >> >> >> >> >> >> >> >> RDMA uverbs layer will not do limit enforcement of HCA hw vendor >> >> >> >> specific resources. Instead rdma cgroup provides set of APIs >> >> >> >> through which vendor specific drivers can do resource accounting >> >> >> >> by making use of rdma cgroup. >> >> >> > >> >> >> > Hi Parav, >> >> >> > I want to propose an extension to the RDMA cgroup which can be done as >> >> >> > follow-up patches. >> >> >> > >> >> >> > Let's add new global type, which will control whole HCA (for example in percentages). It will >> >> >> > allow natively define new objects without need to introduce them to the user. >> >> >> > >> >> >> In other cgroup such as CPU, this is done using cpu.weight API. Where >> >> >> percentage or weight is configured by the user. >> >> >> In this mode, resources taken away from other cgroup proportionately. >> >> >> It works for cpu because its mainly stateless resource unlike rdma >> >> >> resources. >> >> >> So if we want to simplify user configuration similarly, >> >> >> percentage/weight configuration can be extended. >> >> >> This way they need not be introduced to users. >> >> >> I hope your definition of "user" is actual end-user and not rdma cgroup. Right? >> >> > >> >> > Yes, "user" -> "admin". >> >> > I think that percentage is more intuitive to them and will be much easier to >> >> > explain how to use it. I always have in mind "swappiness" field and the >> >> > numerous questions on how to configure it. >> >> > >> >> >> In other words, new object should be still added as new enum value in >> >> >> rdma_cgroup.h? >> >> > >> >> > Yes, I had in mind something like IB_CGROUP_HCA, this is why it can be >> >> > done as a future work after accepting current patches. >> >> > >> >> What I meant is, >> >> today we have RDMACG_VERB_RESOURCE_QP etc, >> >> We will additionally have RDMACG_VERB_RESOURCE_INDIRECT_TBL etc in >> >> cgroup_rdma.h. >> >> So that its available for admin to override it. >> > >> > IMHO, we are talking about the same. My global HCA object will be >> > overwritten by more granular VERBS objects in case they exists. >> > >> >> >> >> >> Only than it can be overwritten by specific UVERBs type as you >> >> >> described below. I think thats what you meant as you described below. >> >> > >> >> > Exactly. >> >> > >> >> >> >> >> >> Otherwise charging/uncharging this new percentage resource can get messy. >> >> > >> >> > Agree >> >> > >> >> >> >> >> >> > This HCA share will be overwritten by specific UVERBS types which you >> >> >> > already defined. >> >> >> > >> >> >> > What do you think? >> >> >> >> >> >> So to refine your proposal from cgroup perspective, instead of adding >> >> >> new resource type in rdma_cgroup.h for percentage, I prefer to have >> >> >> >> >> >> Existing >> >> >> 1. rdma.max >> >> >> 2. rdma.current >> >> >> New, >> >> >> 3. rdma.weight >> >> >> This ABI will have similar API to say >> >> >> echo "mlx4_0 50" > rdma.weight. >> >> >> Where 50 is weight of the resources. >> >> >> For example, >> >> >> for one cgroup instance weight=sum=100% resource for a given cgroup. >> >> >> for three cgroup instances percentage=(weight/sum)% = 50/(50+50+50) = 33%. >> >> >> One cgroup gets 33% resource. >> >> >> >> >> >> Weight can be in range of 1 to 10,000 similar to cpu cgroup. >> >> > >> >> > This is exactly what I don't like, the percentage will remove from the >> >> > user the translation needs between weight and actual limitation. >> >> > >> >> > IMHO CPU used weights because everything there is in weights :). >> >> > >> >> I admit weight are not very intuitive, I was aligning to the existing >> >> other cgroup interfaces which achieves similar functionality. >> >> I will let Tejun approve the "percentage" or "ratio" new file >> >> interface as its little different than weight. >> > >> > Sure, let's close the main idea first and see if it makes sense for >> > other participants. >> > >> >> >> >> >> >> >> >> This might work if applications running in all cgroups are similar. >> >> >> But weight doesn't do justice, when there are different type of >> >> >> applications running in each cgroup. Such as few running libfabric >> >> >> based apps, few running MPI, others directly using ibverbs. >> >> >> So as you said rdma.max configuration would be required for management >> >> >> plane to override weight (percentage) for certain resources. >> >> > >> >> > Why? >> >> > The device exposes max values during initialization and if user asked >> >> > for 20% percent of HCA, he will get max*0.2. >> >> >> >> Because every application may not be equivalent of other application. >> >> For example, some require one to one QP and PD mapping. >> >> Some share single PD across multiple QPs. >> >> Some have ratio of 100 MRs per QP, as factor of memory size and operations. >> >> some servers like to have 1K MRs per QP. >> >> So if we have just weight, it will equally distributes MRs per QP in >> >> all cgroup and that either leads to unused resource per cgroup or, >> >> lesser number of cg instances. >> >> So fine tuning required for individual one, which we already have. >> > >> > I afraid that it is over complicating which can be done by curious user >> > in his user-space scripts: limit the global HCA -> read max values -> >> > overwrite with specific mapping. >> > >> >> >> >> weight or percentage helps in abstracting as starting point. So I like >> >> to add it too. >> > >> > Let's start simple >> >> Yes. I will rebase and test my patch today and see if requires resending. > > It is worth to wait till -rc1. Doug didn't finish his pull requests yet. > Alright. I will wait. > Thanks -- To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
