On Wed, Dec 4, 2019 at 2:54 PM Felix Abecassis <fabecassis@xxxxxxxxxx> wrote:
>
> On Wed, Dec 04, 2019 at 12:17:59PM -0800, Yang Shi wrote:
> > On Wed, Dec 4, 2019 at 11:01 AM Felix Abecassis <fabecassis@xxxxxxxxxx> wrote:
> > >
> > > Hello all,
> > >
> > > On kernel 5.3, when using the move_pages syscall (wrapped by libnuma) and all
> > > pages happen to be on the right node already, this function returns 0 but the
> > > "status" array is not updated. This array potentially contains garbage values
> > > (e.g. from malloc(3)), and I don't see a way to detect this.
> > >
> > > Looking at the kernel code, we are probably exiting do_pages_move here:
> > > out_flush:
> > > if (list_empty(&pagelist))
> > > return err;
> >
> > May you please give the below patch a try? I just did build test.
> >
> > diff --git a/mm/migrate.c b/mm/migrate.c
> > index a8f87cb..f2f1279 100644
> > --- a/mm/migrate.c
> > +++ b/mm/migrate.c
> > @@ -1517,7 +1517,8 @@ static int do_move_pages_to_node(struct mm_struct *mm,
> > * the target node
> > */
> > static int add_page_for_migration(struct mm_struct *mm, unsigned long addr,
> > - int node, struct list_head *pagelist, bool migrate_all)
> > + int node, struct list_head *pagelist, bool migrate_all,
> > + int __user *status, int start)
> > {
> > struct vm_area_struct *vma;
> > struct page *page;
> > @@ -1543,8 +1544,10 @@ static int add_page_for_migration(struct
> > mm_struct *mm, unsigned long addr,
> > goto out;
> >
> > err = 0;
> > - if (page_to_nid(page) == node)
> > + if (page_to_nid(page) == node) {
> > + err = store_status(status, start, node, 1);
> > goto out_putpage;
> > + }
> >
> > err = -EACCES;
> > if (page_mapcount(page) > 1 && !migrate_all)
> > @@ -1639,7 +1642,9 @@ static int do_pages_move(struct mm_struct *mm,
> > nodemask_t task_nodes,
> > * report them via status
> > */
> > err = add_page_for_migration(mm, addr, current_node,
> > - &pagelist, flags & MPOL_MF_MOVE_ALL);
> > + &pagelist, flags & MPOL_MF_MOVE_ALL, status,
> > + i);
> > +
> > if (!err)
> > continue;
> >
>
> Thanks! I have tested this patch on top of 5.4-rc8, and I can comfirm it fixes
> this particular test case.
Thanks for confirming this. I will send out formal patch soon.
>
> > >
> > > Here is a sample C program to reproduce the problem:
> > > /* $ gcc move_pages_bug.c -lnuma -o move_pages_bug */
> > >
> > > #define _DEFAULT_SOURCE
> > > #include <sys/mman.h>
> > >
> > > #include <numaif.h>
> > > #include <stdlib.h>
> > > #include <stdio.h>
> > > #include <unistd.h>
> > >
> > > int main(void)
> > > {
> > > const long node_id = 1;
> > > const long page_size = sysconf(_SC_PAGESIZE);
> > > const int64_t num_pages = 8;
> > >
> > > unsigned long nodemask = 1 << node_id;
> > > long ret = set_mempolicy(MPOL_BIND, &nodemask, sizeof(nodemask));
> > > if (ret < 0)
> > > return (EXIT_FAILURE);
> > >
> > > void **pages = malloc(sizeof(void*) * num_pages);
> > > for (int i = 0; i < num_pages; ++i) {
> > > pages[i] = mmap(NULL, page_size, PROT_WRITE | PROT_READ,
> > > MAP_PRIVATE | MAP_POPULATE | MAP_ANONYMOUS,
> > > -1, 0);
> > > if (pages[i] == MAP_FAILED)
> > > return (EXIT_FAILURE);
> > > }
> > >
> > > ret = set_mempolicy(MPOL_DEFAULT, NULL, 0);
> > > if (ret < 0)
> > > return (EXIT_FAILURE);
> > >
> > > int *nodes = malloc(sizeof(int) * num_pages);
> > > int *status = malloc(sizeof(int) * num_pages);
> > > for (int i = 0; i < num_pages; ++i) {
> > > nodes[i] = node_id;
> > > status[i] = 0xd0; /* simulate garbage values */
> > > }
> > >
> > > ret = move_pages(0, num_pages, pages, nodes, status, MPOL_MF_MOVE);
> > > printf("move_pages: %ld\n", ret);
> > > for (int i = 0; i < num_pages; ++i)
> > > printf("status[%d] = %d\n", i, status[i]);
> > > }
> > >
> > > And here is a sample output showing the "garbage" values:
> > > $ ./move_pages_bug
> > > move_pages: 0
> > > status[0] = 208
> > > status[1] = 208
> > > status[2] = 208
> > > status[3] = 208
> > > status[4] = 208
> > > status[5] = 208
> > > status[6] = 208
> > > status[7] = 208
> > >
> > > Note that passing NULL as the "nodes" argument works as expected here.
> > >
> > > Also, it seems that it's the last "run-length" of pages on the right node(s)
> > > that triggers this problem, e.g. if I add "nodes[0] = nodes[1] = 0", then the
> > > output becomes:
> > > $ ./move_pages_bug
> > > move_pages: 0
> > > status[0] = 0
> > > status[1] = 0
> > > status[2] = 208
> > > status[3] = 208
> > > status[4] = 208
> > > status[5] = 208
> > > status[6] = 208
> > > status[7] = 208
> > >
> > > And with just "nodes[7] = 0;", the first run-length of pages gets assigned
> > > correctly:
> > > $ ./move_pages_bug
> > > move_pages: 0
> > > status[0] = 1
> > > status[1] = 1
> > > status[2] = 1
> > > status[3] = 1
> > > status[4] = 1
> > > status[5] = 1
> > > status[6] = 1
> > > status[7] = 0
> > >
> > >
> > > Thank you,
> > >
