move_pages() blocks sometimes with SCHED_FIFO thread bound to CPU core

[Martin Kulas <kulas@xxxxxxx>]

Hello Linux NUMA experts,


I have already asked this question on SO but no answere there:
https://stackoverflow.com/questions/53725068/move-pages-blocks-sometimes-when-real-time-priority-thread-bound-to-one-core
Therefore,  I will give it a try here.

The problem is reduced to the program numa_rt_prio.c which
starts a background thread with policy SCHED_FIFO and real-time priority
on CPU core 4 and,  afterwards,  the program moves one memory page
to NUMA node 0:

<begin>
#define _GNU_SOURCE
#include <assert.h>
#include <numa.h>
#include <numaif.h>
#include <pthread.h>
#include <sched.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

const int MY_POLICY= SCHED_FIFO;
const int MY_PRIORITY= 10;

struct ThreadCtrl
{
    volatile int terminationRequested;
};

static void *startRoutine(void *args)
{
    struct ThreadCtrl *ctrl= (struct ThreadCtrl *) args;

    while (! ctrl->terminationRequested)
    {
    }

    return 0;
}

int main(int argc, char **argv)
{
    int error= 0;

    pthread_attr_t attr;
    error= pthread_attr_init(&attr);
    assert(error == 0);

    const int THREAD_CORE= 4;
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(THREAD_CORE, &cpuset);
    error= pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &cpuset);
    assert(error == 0);

    error= pthread_attr_setschedpolicy(&attr, MY_POLICY);
    assert(error == 0);

    error= pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
    assert(error == 0);

    struct sched_param threadParam;
    error= pthread_attr_getschedparam(&attr, &threadParam);
    assert(error == 0);

    threadParam.sched_priority= MY_PRIORITY;
    error= pthread_attr_setschedparam(&attr, &threadParam);
    assert(error == 0);

    pthread_t thread;
    struct ThreadCtrl threadCtrl= {.terminationRequested = 0};
    error= pthread_create(&thread, &attr, startRoutine, &threadCtrl);
    assert(error == 0);

    printf("Sleeping and waiting until thread runs ...\n");
    sleep(3);
    printf("Awaked\n");

    const size_t BUF_SIZE= 42;
    void *ptr;
    error= posix_memalign(&ptr, sysconf(_SC_PAGE_SIZE), BUF_SIZE);
    assert(error == 0);
    char *buf= ptr;
    memset(buf, 0, BUF_SIZE);

    int pageNode= 0;
    int status= -1;
    ptr= buf;
    printf("Moving one page ...\n");
    error= move_pages(0, 1, &ptr, &pageNode, &status, MPOL_MF_MOVE);
    assert(error == 0);
    assert(status == pageNode);
    printf("One page has been moved ...\n");

    threadCtrl.terminationRequested= 1;
    error= pthread_join(thread, 0);
    assert(error == 0);

    return EXIT_SUCCESS;
}

<end>

After compiling with 
`cc -Wall -o numa_rt_prio -pthread numa_rt_prio.c -lnuma`, 
an invocation (of course as root) sometimes blocks:

    # ./numa_rt_prio
    Sleeping and waiting until thread runs ...
    Awaked
    Moving one page ...

But sometimes, the program runs without any problem:

    # ./numa_rt_prio 
    Sleeping and waiting until thread runs ...
    Awaked
    Moving one page ...
    One page has been moved ...

If the background thread runs without real-time priorities, move_pages()
always succeeds. If the background thread is not bound to any CPU core,
move_pages() always succeeds as well.

Even decreasing sched_rt_runtime_us does not help:

    $ cat /proc/sys/kernel/sched_rt_runtime_us 
    550000

Do you have any idea what the reason for the blocking move_pages() is?


Kind regards!
Martin