Report a huge page issue in kernel version v5.19.xx

[Fumin Gao <Fumin.Gao@xxxxxxxxxxxxxxxxxx>]

Hi,

 

What’s the issue?

Recently in our product, I found a issue in kernel version v5.19.xx, this issue was fixed in kernel version v6.xx.

The issue is I can’t get which node the huge page is on by system call move_pages.

 

How to reproduce this issue? 

I attached my test programme file in email.

virtaddr = mmap(NULL, ONE_GIG, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB , -1, 0);

*(char *)virtaddr = 0;

if (syscall(SYS_move_pages, 0, 1, &virtaddr, NULL, &NumaNode, 0) != 0)

{

   printf("Get virtual address 0x%p on NumaNode failed \n", virtaddr);

}

printf("create shared memory with mmap, virtaddr 0x%lx on Node %d, errno %d \n", virtaddr,NumaNode, errno);

 

When tested with kernel v5.19.xx , the value of  NumaNode is -2 (-ENOENT).

 

My analysis of this issue.

Based on the following trace and kernel source code, I can see the function calling process.

kernel_move_pages – do_pages_stat –  do_pages_stat_array — follow_page — follow_page_mask

— follow_p4d_mask — follow_pud_mask — follow_huge_pud

 

[001] ..... 510329749178328: sys_move_pages(pid: 0, nr_pages: 1, pages: 7fffa23a2c90, nodes: 0, status: 7fffa23a2c9c, flags: 0)

[001] ..... 510329749179360: sys_enter: NR 279 (0, 1, 7fffa23a2c90, 0, 7fffa23a2c9c, 0)

[001] ...1. 510329749185448: mmap_lock_start_locking: mm=00000000e0f35bcd memcg_path=/user.slice/user-1000.slice/session-1.scope write=false

[001] ...1. 510329749187872: mmap_lock_acquire_returned: mm=00000000e0f35bcd memcg_path=/user.slice/user-1000.slice/session-1.scope write=false success=true

[001] ..... 510329749196628: p_follow_page_0: (follow_page+0x0/0xe0)

[001] ..... 510329749199690: p_vma_is_secretmem_0: (vma_is_secretmem+0x0/0x20)

[001] ..... 510329749202194: p_follow_page_mask_0: (follow_page_mask+0x0/0x160)

[001] ..... 510329749206928: p_follow_huge_addr_0: (follow_huge_addr+0x0/0x20)

[001] ..... 510329749210628: myretprobe: (follow_page_mask+0x38/0x160 <- follow_huge_addr) ret=0xffffffffffffffea

[001] ..... 510329749216464: p_follow_pud_mask_isra_0_0: (follow_pud_mask.isra.0+0x0/0x1e0)

[001] ..... 510329749221108: p_follow_huge_pud_0: (follow_huge_pud+0x0/0x80)

[001] ..... 510329749221902: myretprobe: (follow_pud_mask.isra.0+0x1c8/0x1e0 <- follow_huge_pud) ret=0x0

[001] ..... 510329749223462: myretprobe: (follow_page_mask+0x147/0x160 <- follow_pud_mask.isra.0) ret=0x0

[001] ..... 510329749224838: myretprobe: (do_pages_stat+0x18b/0x330 <- follow_page) ret=0x0

[001] ...1. 510329749226096: mmap_lock_released: mm=00000000e0f35bcd memcg_path=/user.slice/user-1000.slice/session-1.scope write=false

[001] ..... 510329749228348: sys_move_pages -> 0x0

[001] ..... 510329749229224: sys_exit: NR 279 = 0

 

In the kernel version v5.19.xx, it add a flag FOLL_GET  in do_pages_stat_array compared with v5.18.xx.

        page = follow_page(vma, addr, FOLL_GET | FOLL_DUMP);

 

But in the function follow_huge_pud, if the flags has FOLL_GET, it will return NULL. This causes we get the status is -ENOENT (-2) 

in move_pages.

 

 

Is my analysis correct ?

 

Thanks

Fumin

 

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
#include <errno.h>
#include <string.h>
#include <signal.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <fcntl.h>

#define ONE_MEG               (1024UL*1024UL)
#define ONE_GIG               (1024UL * ONE_MEG)

int main(int argc, char * argv[])
{
	char s[32];
	int  NumaNode;
    void *virtaddr;
	virtaddr = mmap(NULL, ONE_GIG, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB , -1, 0);
    if (virtaddr == -1)
    {
	   printf("mmap return failed\n");
       perror("mmap");
       exit(1);
    }
	*(char *)virtaddr = 0;
	
	if (syscall(SYS_move_pages, 0, 1, &virtaddr, NULL, &NumaNode, 0) != 0)
    {
       printf("Get virtual address 0x%p on NumaNode failed \n", virtaddr);
    }
	
	printf("create shared memory with mmap, virtaddr 0x%lx on Node %d, errno %d \n", virtaddr,NumaNode, errno);
    
	return 0;
}