On Tue, Jul 12, 2022 at 10:56:11AM -0400, Joe Lawrence wrote:
> On Thu, Jun 30, 2022 at 11:12:26AM -0300, Marcos Paulo de Souza wrote:
> > Syscalls are called a tricky way. Some architectures add a prefix to the
> > syscall name (SYSCALL_WRAPPER).
> >
> > This new test creates one userspace process per online cpu calling getpid
> > continuously and tries to livepatch the getpid function. Add the correct
> > function prefix for all archs that select HAS_SYSCALL_WRAPPER.
> >
> > Signed-off-by: Marcos Paulo de Souza <mpdesouza@xxxxxxxx>
>
> Hi Marcos,
>
> Thanks for working on this and sorry for the delayed reply. I gave it a
> spin yesterday and have a few comments below...
It has been sometime since the last version, thanks for the comments. I have
some questions bellow.
>
> > ---
> > tools/testing/selftests/livepatch/Makefile | 12 +-
> > .../selftests/livepatch/test-syscall.sh | 52 ++++++
> > .../test_binaries/test_klp-call_getpid.c | 48 ++++++
> > .../selftests/livepatch/test_modules/Makefile | 3 +-
> > .../livepatch/test_modules/test_klp_syscall.c | 150 ++++++++++++++++++
> > 5 files changed, 261 insertions(+), 4 deletions(-)
> > create mode 100755 tools/testing/selftests/livepatch/test-syscall.sh
> > create mode 100644 tools/testing/selftests/livepatch/test_binaries/test_klp-call_getpid.c
> > create mode 100644 tools/testing/selftests/livepatch/test_modules/test_klp_syscall.c
> >
> > diff --git a/tools/testing/selftests/livepatch/Makefile b/tools/testing/selftests/livepatch/Makefile
> > index 5ef492b87bb1..35014197184e 100644
> > --- a/tools/testing/selftests/livepatch/Makefile
> > +++ b/tools/testing/selftests/livepatch/Makefile
> > @@ -1,10 +1,14 @@
> > # SPDX-License-Identifier: GPL-2.0
> > +include ../../../build/Build.include
> > +include ../../../scripts/Makefile.arch
> > +include ../../../scripts/Makefile.include
> >
> > TEST_FILES := settings \
> > test_modules
> >
> > # We need the test_modules dir in order to make gen_tar and install to work
> > -TEST_GEN_PROGS_EXTENDED := test_modules/test_klp_atomic_replace.ko \
> > +TEST_GEN_PROGS_EXTENDED := test_binaries/test_klp-call_getpid \
>
> Before I commit this to muscle memory, would "test_programs" better
> describe this directory? When I see "binaries", I think of source vs.
> build and not kernel vs. userspace.
Done locally.
>
> > + test_modules/test_klp_atomic_replace.ko \
> > test_modules/test_klp_callbacks_busy.ko \
> > test_modules/test_klp_callbacks_demo.ko \
> > test_modules/test_klp_callbacks_demo2.ko \
> > @@ -13,7 +17,8 @@ TEST_GEN_PROGS_EXTENDED := test_modules/test_klp_atomic_replace.ko \
> > test_modules/test_klp_state.ko \
> > test_modules/test_klp_state2.ko \
> > test_modules/test_klp_state3.ko \
> > - test_modules/test_klp_shadow_vars.ko
> > + test_modules/test_klp_shadow_vars.ko \
> > + test_modules/test_klp_syscall.ko
> >
> > TEST_PROGS_EXTENDED := functions.sh
> > TEST_PROGS := \
> > @@ -21,7 +26,8 @@ TEST_PROGS := \
> > test-callbacks.sh \
> > test-shadow-vars.sh \
> > test-state.sh \
> > - test-ftrace.sh
> > + test-ftrace.sh \
> > + test-syscall.sh
> >
> > # override lib.mk's default rules
> > OVERRIDE_TARGETS := 1
> > diff --git a/tools/testing/selftests/livepatch/test-syscall.sh b/tools/testing/selftests/livepatch/test-syscall.sh
> > new file mode 100755
> > index 000000000000..fb3270de7f1f
> > --- /dev/null
> > +++ b/tools/testing/selftests/livepatch/test-syscall.sh
> > @@ -0,0 +1,52 @@
> > +#!/bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (C) 2022 SUSE
> > +# Author: Marcos Paulo de Souza <mpdesouza@xxxxxxxx>
> > +
> > +. $(dirname $0)/functions.sh
> > +
> > +MOD_SYSCALL=test_klp_syscall
> > +
> > +setup_config
> > +
> > +# - Start _NRPROC processes calling getpid and load a livepatch to patch the
> > +# getpid syscall
> > +
> > +start_test "patch getpid syscall while being heavily hammered"
> > +
> > +for i in $(seq 0 $(( $(getconf _NPROCESSORS_ONLN) -1)) ); do
> > + ./test_binaries/test_klp-call_getpid &
> > + pids[$i]="$!"
> > +done
>
> On second thought, bash should only iterate through the elements that
> are defined, so your earlier version is probably better:
>
> for i in $(seq 1 $(getconf _NPROCESSORS_ONLN)); do
>
> > +
> > +pid_list=$(echo ${pids[@]} | tr ' ' ',')
> > +load_lp $MOD_SYSCALL klp_pids=$pid_list
> > +
>
> We could wait for all test_klp-call_getpid instances to execute the
> livepatch code:
>
> loop_until 'grep -q '^0$' /sys/kernel/test_klp_syscall/npids'
>
> This has a built in timeout, so it's not infinitely looping. You
> could add some die "reason" logic here, or just let the existing code
> cleanup after the failed test.
Makes sense. Done locally.
>
> > +pending_pids=$(cat /sys/kernel/test_klp_syscall/npids)
> > +
> > +for pid in ${pids[@]}; do
> > + kill $pid || true
> > +done
> > +
> > +disable_lp $MOD_SYSCALL
> > +unload_lp $MOD_SYSCALL
> > +
> > +if [[ "$pending_pids" != "0" ]]; then
> > + echo -e "FAIL\n\n"
> > + die "processes not livepatched: $pending_pids"
> > +fi
> > +
> > +check_result "% insmod test_modules/$MOD_SYSCALL.ko klp_pids=$pid_list
> > +livepatch: enabling patch '$MOD_SYSCALL'
> > +livepatch: '$MOD_SYSCALL': initializing patching transition
> > +livepatch: '$MOD_SYSCALL': starting patching transition
> > +livepatch: '$MOD_SYSCALL': completing patching transition
> > +livepatch: '$MOD_SYSCALL': patching complete
> > +% echo 0 > /sys/kernel/livepatch/$MOD_SYSCALL/enabled
> > +livepatch: '$MOD_SYSCALL': initializing unpatching transition
> > +livepatch: '$MOD_SYSCALL': starting unpatching transition
> > +livepatch: '$MOD_SYSCALL': completing unpatching transition
> > +livepatch: '$MOD_SYSCALL': unpatching complete
> > +% rmmod $MOD_SYSCALL"
> > +
> > +exit 0
> > diff --git a/tools/testing/selftests/livepatch/test_binaries/test_klp-call_getpid.c b/tools/testing/selftests/livepatch/test_binaries/test_klp-call_getpid.c
> > new file mode 100644
> > index 000000000000..ec470660dee6
> > --- /dev/null
> > +++ b/tools/testing/selftests/livepatch/test_binaries/test_klp-call_getpid.c
> > @@ -0,0 +1,48 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Copyright (C) 2017 SUSE
> > + * Author: Libor Pechacek <lpechacek@xxxxxxx>
> > + */
> > +
> > +#include <stdio.h>
> > +#include <unistd.h>
> > +#include <sys/syscall.h>
> > +#include <sys/types.h>
> > +#include <signal.h>
> > +
> > +static int stop = 0;
>
> nit: no need to init global to 0
Indeed. Fixed.
>
> > +static int sig_int;
> > +
> > +void hup_handler(int signum)
> > +{
> > + stop = 1;
> > +}
> > +
> > +void int_handler(int signum)
> > +{
> > + stop = 1;
> > + sig_int = 1;
> > +}
> > +
> > +int main(int argc, char *argv[])
> > +{
> > + pid_t orig_pid, pid;
> > + long count = 0;
> > +
> > + signal(SIGHUP, &hup_handler);
> > + signal(SIGINT, &int_handler);
> > +
> > + orig_pid = syscall(SYS_getpid);
> > +
> > + while (!stop) {
> > + pid = syscall(SYS_getpid);
> > + if (pid != orig_pid)
> > + return 1;
>
> This test doesn't care about the user program return code, but I wonder
> if the status should be flipped -- this is the desired code path, not
> the one at the end of main(), right?
I got the code from our tests. IIUC, syscall with SYS_getpid cannot fail. I'll
double check before sending v3.
>
> > + count++;
> > + }
> > +
> > + if (sig_int)
> > + printf("%ld iterations done\n", count);
> > +
> > + return 0;
> > +}
> > diff --git a/tools/testing/selftests/livepatch/test_modules/Makefile b/tools/testing/selftests/livepatch/test_modules/Makefile
> > index 1eab43b741cd..ebb754accf46 100644
> > --- a/tools/testing/selftests/livepatch/test_modules/Makefile
> > +++ b/tools/testing/selftests/livepatch/test_modules/Makefile
> > @@ -10,7 +10,8 @@ obj-m += test_klp_atomic_replace.o \
> > test_klp_state.o \
> > test_klp_state2.o \
> > test_klp_state3.o \
> > - test_klp_shadow_vars.o
> > + test_klp_shadow_vars.o \
> > + test_klp_syscall.o
> >
> > %.ko:
> > make -C $(KDIR) M=$(TESTMODS_DIR) $@
> > diff --git a/tools/testing/selftests/livepatch/test_modules/test_klp_syscall.c b/tools/testing/selftests/livepatch/test_modules/test_klp_syscall.c
> > new file mode 100644
> > index 000000000000..e90f4ac8e7a4
> > --- /dev/null
> > +++ b/tools/testing/selftests/livepatch/test_modules/test_klp_syscall.c
> > @@ -0,0 +1,150 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Copyright (C) 2017-2022 SUSE
> > + * Authors: Libor Pechacek <lpechacek@xxxxxxx>
> > + * Nicolai Stange <nstange@xxxxxxx>
> > + * Marcos Paulo de Souza <mpdesouza@xxxxxxxx>
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/kernel.h>
> > +#include <linux/sched.h>
> > +#include <linux/slab.h>
> > +#include <linux/livepatch.h>
> > +
> > +#if defined(__x86_64__)
> > +#define FN_PREFIX __x64_
> > +#elif defined(__s390x__)
> > +#define FN_PREFIX __s390x_
> > +#elif defined(__aarch64__)
> > +#define FN_PREFIX __arm64_
> > +#else
> > +/* powerpc does not select ARCH_HAS_SYSCALL_WRAPPER */
> > +#define FN_PREFIX
> > +#endif
> > +
> > +struct klp_pid_t {
> > + pid_t pid;
> > + struct list_head list;
> > +};
> > +static LIST_HEAD(klp_pid_list);
> > +
> > +/* Protects klp_pid_list */
> > +static DEFINE_MUTEX(kpid_mutex);
> > +
> > +static int klp_pids[NR_CPUS];
> > +static unsigned int npids;
> > +module_param_array(klp_pids, int, &npids, 0);
> > +
> > +static ssize_t npids_show(struct kobject *kobj, struct kobj_attribute *attr,
> > + char *buf)
> > +{
> > + return sprintf(buf, "%u\n", npids);
> > +}
> > +
> > +static struct kobj_attribute klp_attr = __ATTR_RO(npids);
> > +static struct kobject *klp_kobj;
> > +
> > +static void free_klp_pid_list(void)
> > +{
> > + struct klp_pid_t *kpid, *temp;
> > +
> > + mutex_lock(&kpid_mutex);
> > + list_for_each_entry_safe(kpid, temp, &klp_pid_list, list) {
> > + list_del(&kpid->list);
> > + kfree(kpid);
> > + }
> > + mutex_unlock(&kpid_mutex);
> > +}
> > +
> > +asmlinkage long lp_sys_getpid(void)
> > +{
> > + struct klp_pid_t *kpid, *temp;
> > +
> > + /*
> > + * For each thread calling getpid, check if the pid exists in
> > + * klp_pid_list. If yes, decrement the npids variable and remove the pid
> > + * from the list. npids will be later used to ensure that all pids
> > + * transitioned to the liveaptched state.
>
> I go back and forth with this comment: 1) the code is pretty
> self-explanatory and 2) it explains what test-syscall.sh is doing with
> npids
>
> Describing the behavioral difference I suggest below would be helpful
> (if you agree w/the change). Something about removing the current pid
> from the list, okay, too.
Makes sense. I'll move the comment to the test.
>
> Then as far as describing npids' purpose, I'd put that up at the top of
> the file where npids is defined. It can be as generic as describing
> what it's counting.
Thanks for the suggestion. A short description will be added when defining the
module argument.
>
> > + */
> > + mutex_lock(&kpid_mutex);
> > + list_for_each_entry_safe(kpid, temp, &klp_pid_list, list) {
> > + if (current->pid == kpid->pid) {
> > + list_del(&kpid->list);
> > + kfree(kpid);
> > + npids--;
> > + break;
>
> I think it would be safer to return task_tgid_vnr() here, but ...
>
> > + }
> > + }
> > + mutex_unlock(&kpid_mutex);
> > +
> > + return task_tgid_vnr(current);
>
> task_pid_vnr() here. That way we're only changing behavior for the
> processes in the list and not all programs across the system.
So, looking at getpid syscall definition, it calls task_tgid_vnr:
SYSCALL_DEFINE0(getpid)
{
return task_tgid_vnr(current);
}
The function task_pid_vnr is called by gettid. So, IIUC, the test should call
task_tgid_vnr, since getpid is being livepatched.
Am I missing something?
>
> > +}
> > +
> > +static struct klp_func vmlinux_funcs[] = {
> > + {
> > + .old_name = __stringify(FN_PREFIX) "sys_getpid",
> > + .new_func = lp_sys_getpid,
> > + }, {}
> > +};
> > +
> > +static struct klp_object objs[] = {
> > + {
> > + /* name being NULL means vmlinux */
> > + .funcs = vmlinux_funcs,
> > + }, {}
> > +};
> > +
> > +static struct klp_patch patch = {
> > + .mod = THIS_MODULE,
> > + .objs = objs,
> > +};
> > +
> > +static int livepatch_init(void)
> > +{
> > + int ret;
> > + struct klp_pid_t *kpid;
> > +
> > + if (npids > 0) {
> > + int i;
> > +
> > + for (i = 0; i < npids; i++) {
> > + kpid = kmalloc(sizeof(struct klp_pid_t), GFP_KERNEL);
> > + if (!kpid)
> > + goto err_mem;
> > +
> > + kpid->pid = klp_pids[i];
> > + list_add(&kpid->list, &klp_pid_list);
> > + }
> > + }
> > +
> > + klp_kobj = kobject_create_and_add("test_klp_syscall", kernel_kobj);
> > + if (!klp_kobj)
> > + goto err_mem;
> > +
> > + ret = sysfs_create_file(klp_kobj, &klp_attr.attr);
> > + if (ret) {
> > + kobject_put(klp_kobj);
> > + goto out_klp_pid_list;
> > + }
> > +
> > + return klp_enable_patch(&patch);
> > +
> > +err_mem:
> > + ret = -ENOMEM;
> > +out_klp_pid_list:
> > + free_klp_pid_list();
> > +
> > + return ret;
> > +}
> > +
> > +static void livepatch_exit(void)
> > +{
> > + free_klp_pid_list();
> > + kobject_put(klp_kobj);
> > +}
> > +
> > +module_init(livepatch_init);
> > +module_exit(livepatch_exit);
> > +MODULE_LICENSE("GPL");
> > +MODULE_INFO(livepatch, "Y");
>
> The linked list code looks like it should work and cleans up properly,
> but if the current test is the extent of what we need, it would be
> simpler to directly use klp_pids[] to track progress: since we're never
> going to livepatch pid 0, we could just search klp_pids[] for our pid
> and then zero it out once we've executed the livepatch code.
>
> This would let us drop out the kmalloc and list management code (looks
> to be nearly 25% of the file at this point). I think the kpid_mutex
> still needs to stay though. Iterating through the fixed array shouldn't
> be too much more expensive than through the (shrinking) klp_pid_list.
>
> On the other hand, if you think that we'll need to adapt this module to
> handle a dynamically changing list of klp_pids[], then perhaps a
> separate list is the way to go.
My first idea was to use klp_pids indexed by NR_CPUS, but I thought about bigger
machines with more CPUs, making the array too big for just a couple of pids to
be tracked...
Thinking again, I don't think it's a problem. Also, I don't believe that we
would need to dynamically add pids, so this code can go. In the next version
I'll just access klp_pids directly. Thanks for the suggestion!
>
> Final note, kbuild is probably my weakest area, so I'm of limited help
> review-wise there. In trying out the code, I wasn't sure exactly how to
> run things. I did:
>
> $ make -C tools/testing/selftests/livepatch/ run_tests
>
> and that built everything and ran the tests (good). I changed a few .c
> files and tried again, but they didn't rebuild. Then from
> tools/testing/selftests/livepatch/test_modules/ I ran `make` and it only
> ran the clean target (not good, I think).
I need to test it and make sure the code is rebuilt. Thanks for noticing this
problem.
>
> What is the typical workflow supposed to be in this combined setup?
The plan was to support all forms of running the selftests: by running it in
place and by running the "installed" kselftests. I'll test it again to make sure
that the code is rebuilt when it should.
>
> Thanks,
>
> --
> Joe
>
