[PATCH v11 00/26] Speculative page faults

[Vinayak Menon <vinmenon@xxxxxxxxxxxxxx>]

Hi Laurent,

We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).

Steps:

1) Run the program

2) The 2 threads will try lock/unlock and prints messages, and main thread does fork. In around 1 minute time, lock/unlock threads will hang on pthread_mutex_lock.


Initially the issue was reported when ptrace was tried on apps. Later it was discovered that the write protect done by fork is causing the issue and the below test code was created.


Observations:

1) We have tried disabling SPF and the issue disappears.

2) Adding this piece of code in __handle_speculative_fault also fixes the problem.

if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte))

    return VM_FAULT_RETRY;

3) As an experiment we tried encapsulating handle_speculative_fault with down_read(mmap_sem) and that too fixes the problem.

4) It is observed that while in wp_page_copy, the contents of the old_page changes which should not ideally happen as the pte is !pte_write.

5) To prove that it is a race, we tried affining the threads to single core, and the issue disappears.

Let us know if you want us to try out any experiments.

Thanks,
Vinayak

/**test.c***/

#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>

#define UNUSED(expr) do { (void)(expr); } while(0)
#define print printf

pthread_mutex_t m;
pthread_cond_t c;

static void* cond_do(void* arg)
{
  UNUSED(arg);
  do {
    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
    pthread_mutex_lock(&m);
 
    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
    pthread_mutex_unlock(&m);
    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
  } while (true);

  return NULL;
}

static void* sig_do(void* arg)
{
  UNUSED(arg);
  do {
    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
    pthread_mutex_lock(&m);

    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
    pthread_mutex_unlock(&m);
    print("%s:%d state=%u addr: %lx\n", __func__, __LINE__, *(uint32_t*)(&m), (unsigned long)&m);
  } while (true);
  return NULL;
}

int main()
{
  pthread_t sig;
  pthread_t cond;
  pthread_mutex_init(&m, NULL);
 
  pthread_create(&cond, NULL, cond_do, NULL);
  pthread_create(&sig, NULL, sig_do, NULL);

  while(1) {
      if (!fork()) {
          usleep(500);
          abort();
      }
      usleep(550);
  }

  pthread_join(sig, NULL);
  pthread_join(cond, NULL);

  return 0;
}