On Fri, Sep 29, 2023 at 11:42:48AM -0700, Andrew Morton wrote: > > The patch titled > Subject: mm: kmem: add direct objcg pointer to task_struct > has been added to the -mm mm-unstable branch. Its filename is > mm-kmem-add-direct-objcg-pointer-to-task_struct.patch > > This patch will shortly appear at > https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-kmem-add-direct-objcg-pointer-to-task_struct.patch > > This patch will later appear in the mm-unstable branch at > git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm > > Before you just go and hit "reply", please: > a) Consider who else should be cc'ed > b) Prefer to cc a suitable mailing list as well > c) Ideally: find the original patch on the mailing list and do a > reply-to-all to that, adding suitable additional cc's > > *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** > > The -mm tree is included into linux-next via the mm-everything > branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm > and is updated there every 2-3 working days > > ------------------------------------------------------ > From: Roman Gushchin <roman.gushchin@xxxxxxxxx> > Subject: mm: kmem: add direct objcg pointer to task_struct > Date: Fri, 29 Sep 2023 11:00:52 -0700 > > To charge a freshly allocated kernel object to a memory cgroup, the kernel > needs to obtain an objcg pointer. Currently it does it indirectly by > obtaining the memcg pointer first and then calling to > __get_obj_cgroup_from_memcg(). > > Usually tasks spend their entire life belonging to the same object cgroup. > So it makes sense to save the objcg pointer on task_struct directly, so > it can be obtained faster. It requires some work on fork, exit and cgroup > migrate paths, but these paths are way colder. > > To avoid any costly synchronization the following rules are applied: > 1) A task sets it's objcg pointer itself. > > 2) If a task is being migrated to another cgroup, the least > significant bit of the objcg pointer is set atomically. > > 3) On the allocation path the objcg pointer is obtained locklessly > using the READ_ONCE() macro and the least significant bit is > checked. If it's set, the following procedure is used to update > it locklessly: > - task->objcg is zeroed using cmpxcg > - new objcg pointer is obtained > - task->objcg is updated using try_cmpxchg > - operation is repeated if try_cmpxcg fails > It guarantees that no updates will be lost if task migration > is racing against objcg pointer update. It also allows to keep > both read and write paths fully lockless. > > Because the task is keeping a reference to the objcg, it can't go away > while the task is alive. > > This commit doesn't change the way the remote memcg charging works. > > Link: https://lkml.kernel.org/r/20230929180056.1122002-3-roman.gushchin@xxxxxxxxx > Signed-off-by: Roman Gushchin (Cruise) <roman.gushchin@xxxxxxxxx> > Cc: David Rientjes <rientjes@xxxxxxxxxx> > Cc: Dennis Zhou <dennis@xxxxxxxxxx> > Cc: Johannes Weiner <hannes@xxxxxxxxxxx> > Cc: Michal Hocko <mhocko@xxxxxxxxxx> > Cc: Muchun Song <muchun.song@xxxxxxxxx> > Cc: Shakeel Butt <shakeelb@xxxxxxxxxx> > Cc: Vlastimil Babka <vbabka@xxxxxxx> > Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> > --- > > include/linux/memcontrol.h | 10 +++ > include/linux/sched.h | 4 + > mm/memcontrol.c | 111 ++++++++++++++++++++++++++++++++--- > 3 files changed, 116 insertions(+), 9 deletions(-) > > --- a/include/linux/memcontrol.h~mm-kmem-add-direct-objcg-pointer-to-task_struct > +++ a/include/linux/memcontrol.h > @@ -544,6 +544,16 @@ static inline bool folio_memcg_kmem(stru > return folio->memcg_data & MEMCG_DATA_KMEM; > } > > +static inline bool current_objcg_needs_update(struct obj_cgroup *objcg) > +{ > + return (struct obj_cgroup *)((unsigned long)objcg & 0x1); > +} > + > +static inline struct obj_cgroup * > +current_objcg_without_update_flag(struct obj_cgroup *objcg) > +{ > + return (struct obj_cgroup *)((unsigned long)objcg & ~0x1); > +} > > #else > static inline bool folio_memcg_kmem(struct folio *folio) > --- a/include/linux/sched.h~mm-kmem-add-direct-objcg-pointer-to-task_struct > +++ a/include/linux/sched.h > @@ -1443,6 +1443,10 @@ struct task_struct { > struct mem_cgroup *active_memcg; > #endif > > +#ifdef CONFIG_MEMCG_KMEM > + struct obj_cgroup *objcg; > +#endif > + > #ifdef CONFIG_BLK_CGROUP > struct gendisk *throttle_disk; > #endif > --- a/mm/memcontrol.c~mm-kmem-add-direct-objcg-pointer-to-task_struct > +++ a/mm/memcontrol.c > @@ -3041,6 +3041,47 @@ static struct obj_cgroup *__get_obj_cgro > return objcg; > } > > +static struct obj_cgroup *current_objcg_update(struct obj_cgroup *old) > +{ > + struct mem_cgroup *memcg; > + struct obj_cgroup *objcg = NULL, *tmp = old; > + > + old = current_objcg_without_update_flag(old); > + if (old) > + obj_cgroup_put(old); > + > + rcu_read_lock(); > + do { > + /* Atomically drop the update bit, */ > + WARN_ON_ONCE(cmpxchg(¤t->objcg, tmp, 0) != tmp); Hi Andrew, can you please, merge a small fixup here? I've got a sparse complaint from the Intel's kernel test robot. Thanks! -- diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 78ab36b5899f..4c762a04a689 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3013,7 +3013,7 @@ static struct obj_cgroup *current_objcg_update(struct obj_cgroup *old) rcu_read_lock(); do { /* Atomically drop the update bit, */ - WARN_ON_ONCE(cmpxchg(¤t->objcg, tmp, 0) != tmp); + WARN_ON_ONCE(cmpxchg(¤t->objcg, tmp, NULL) != tmp); /* ...obtain the new objcg pointer */ memcg = mem_cgroup_from_task(current);
