Hello,
Sorry for the late reply.
Setting TEST_ITEMS="xx yy zz" and invoking the script has the same
effect as passing -t "xx yy zz" to the script. VM_TEST_ITEMS may be a
better env variable name however.
Best,
Joel Savitz
On Fri, Jul 15, 2022 at 3:55 PM Nico Pache <npache@xxxxxxxxxx> wrote:
>
> Hi Andrew and Joel,
>
> I noticed a flaw in switching to the cmdline argument over the
> environment variable. If you are utilizing the exerciser script (in
> tools/testing/selftests/run_kselftest.sh), this is not designed to
> pass cmdline variables to the individual TEST_PROGS. On the other hand
> if we utilize Env. variables we can still specify which tests we'd
> like to run. Our infrastructure is designed to utilize this wrapper,
> so we inevitably will not be able to use this the way we thought.
>
> Would you be ok with switching back to the initial approach? If so I
> believe making the name more specific would be ideal, ie)
> VM_TEST_ITEMS.
>
> Cheers,
> -- Nico
>
> On Wed, Jul 6, 2022 at 4:55 PM Joel Savitz <jsavitz@xxxxxxxxxx> wrote:
> >
> > Our memory management kernel CI testing at Red Hat uses the VM
> > selftests and we have run into two problems:
> >
> > First, our LTP tests overlap with the VM selftests.
> >
> > We want to avoid unhelpful redundancy in our testing practices.
> >
> > Second, we have observed the current run_vmtests.sh to report overall
> > failure/ambiguous results in the case that a machine lacks the necessary
> > hardware to perform one or more of the tests. E.g. ksm tests that
> > require more than one numa node.
> >
> > We want to be able to run the vm selftests suitable to particular hardware.
> >
> > Add the ability to run one or more groups of vm tests via run_vmtests.sh
> > instead of simply all-or-none in order to solve these problems.
> >
> > Preserve existing default behavior of running all tests when the script
> > is invoked with no arguments.
> >
> > Documentation of test groups is included in the patch as follows:
> >
> > # ./run_vmtests.sh [ -h || --help ]
> >
> > usage: ./tools/testing/selftests/vm/run_vmtests.sh [ -h | -t "<categories>"]
> > -t: specify specific categories to tests to run
> > -h: display this message
> >
> > The default behavior is to run all tests.
> >
> > Alternatively, specific groups tests can be run by passing a string
> > to the -t argument containing one or more of the following categories
> > separated by spaces:
> > - mmap
> > tests for mmap(2)
> > - gup_test
> > tests for gup using gup_test interface
> > - userfaultfd
> > tests for userfaultfd(2)
> > - compaction
> > a test for the patch "Allow compaction of unevictable pages"
> > - mlock
> > tests for mlock(2)
> > - mremap
> > tests for mremap(2)
> > - hugevm
> > tests for very large virtual address space
> > - vmalloc
> > vmalloc smoke tests
> > - hmm
> > hmm smoke tests
> > - madv_populate
> > test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> > - memfd_secret
> > test memfd_secret(2)
> > - process_mrelease
> > test process_mrelease(2)
> > - ksm
> > ksm tests that do not require >=2 NUMA nodes
> > - ksm_numa
> > ksm tests that require >=2 NUMA nodes
> > - pkey
> > memory protection key tests
> > example: ./run_vmtests.sh -t "hmm mmap ksm"
> >
> > Changes from v2:
> > - rebase onto the mm-everyting branch in
> > https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
> > - integrate this functionality with new the tests
> >
> > Changes from v1:
> > - use a command line argument to pass the test categories to the
> > script instead of an environmet variable
> > - remove novel prints to avoid messing with extant parsers of this
> > script
> > - update the usage text
> >
> > Signed-off-by: Joel Savitz <jsavitz@xxxxxxxxxx>
> > ---
> > tools/testing/selftests/vm/run_vmtests.sh | 241 +++++++++++++++-------
> > 1 file changed, 161 insertions(+), 80 deletions(-)
> >
> > diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh
> > index d84fe0fa15e1..fb72a1338d07 100755
> > --- a/tools/testing/selftests/vm/run_vmtests.sh
> > +++ b/tools/testing/selftests/vm/run_vmtests.sh
> > @@ -1,6 +1,6 @@
> > #!/bin/bash
> > # SPDX-License-Identifier: GPL-2.0
> > -#please run as root
> > +# Please run as root
> >
> > # Kselftest framework requirement - SKIP code is 4.
> > ksft_skip=4
> > @@ -8,15 +8,75 @@ ksft_skip=4
> > mnt=./huge
> > exitcode=0
> >
> > -#get huge pagesize and freepages from /proc/meminfo
> > -while read -r name size unit; do
> > - if [ "$name" = "HugePages_Free:" ]; then
> > - freepgs="$size"
> > - fi
> > - if [ "$name" = "Hugepagesize:" ]; then
> > - hpgsize_KB="$size"
> > +usage() {
> > + cat <<EOF
> > +usage: ${BASH_SOURCE[0]:-$0} [ -h | -t "<categories>"]
> > + -t: specify specific categories to tests to run
> > + -h: display this message
> > +
> > +The default behavior is to run all tests.
> > +
> > +Alternatively, specific groups tests can be run by passing a string
> > +to the -t argument containing one or more of the following categories
> > +separated by spaces:
> > +- mmap
> > + tests for mmap(2)
> > +- gup_test
> > + tests for gup using gup_test interface
> > +- userfaultfd
> > + tests for userfaultfd(2)
> > +- compaction
> > + a test for the patch "Allow compaction of unevictable pages"
> > +- mlock
> > + tests for mlock(2)
> > +- mremap
> > + tests for mremap(2)
> > +- hugevm
> > + tests for very large virtual address space
> > +- vmalloc
> > + vmalloc smoke tests
> > +- hmm
> > + hmm smoke tests
> > +- madv_populate
> > + test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> > +- memfd_secret
> > + test memfd_secret(2)
> > +- process_mrelease
> > + test process_mrelease(2)
> > +- ksm
> > + ksm tests that do not require >=2 NUMA nodes
> > +- ksm_numa
> > + ksm tests that require >=2 NUMA nodes
> > +- pkey
> > + memory protection key tests
> > +example: ./run_vmtests.sh -t "hmm mmap ksm"
> > +EOF
> > + exit 0
> > +}
> > +
> > +
> > +while getopts "ht:" OPT; do
> > + case ${OPT} in
> > + "h") usage ;;
> > + "t") TEST_ITEMS=${OPTARG} ;;
> > + esac
> > +done
> > +shift $((OPTIND -1))
> > +
> > +# default behavior: run all tests
> > +TEST_ITEMS=${TEST_ITEMS:-default}
> > +
> > +test_selected() {
> > + if [ "$TEST_ITEMS" == "default" ]; then
> > + # If no TEST_ITEMS are specified, run all tests
> > + return 0
> > fi
> > -done < /proc/meminfo
> > + echo ${TEST_ITEMS} | grep ${1} 2>&1 >/dev/null
> > + return ${?}
> > +}
> > +
> > +# Hugepage setup only needed for hugetlb tests
> > +if test_selected "hugetlb"; then
> >
> > # Simple hugetlbfs tests have a hardcoded minimum requirement of
> > # huge pages totaling 256MB (262144KB) in size. The userfaultfd
> > @@ -28,7 +88,17 @@ hpgsize_MB=$((hpgsize_KB / 1024))
> > half_ufd_size_MB=$((((nr_cpus * hpgsize_MB + 127) / 128) * 128))
> > needmem_KB=$((half_ufd_size_MB * 2 * 1024))
> >
> > -#set proper nr_hugepages
> > +# get huge pagesize and freepages from /proc/meminfo
> > +while read -r name size unit; do
> > + if [ "$name" = "HugePages_Free:" ]; then
> > + freepgs="$size"
> > + fi
> > + if [ "$name" = "Hugepagesize:" ]; then
> > + hpgsize_KB="$size"
> > + fi
> > +done < /proc/meminfo
> > +
> > +# set proper nr_hugepages
> > if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
> > nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
> > needpgs=$((needmem_KB / hpgsize_KB))
> > @@ -57,144 +127,155 @@ else
> > exit 1
> > fi
> >
> > -#filter 64bit architectures
> > +fi # test_selected "hugetlb"
> > +
> > +# filter 64bit architectures
> > ARCH64STR="arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64"
> > if [ -z "$ARCH" ]; then
> > ARCH=$(uname -m 2>/dev/null | sed -e 's/aarch64.*/arm64/')
> > fi
> > VADDR64=0
> > -echo "$ARCH64STR" | grep "$ARCH" && VADDR64=1
> > +echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
> >
> > # Usage: run_test [test binary] [arbitrary test arguments...]
> > run_test() {
> > - local title="running $*"
> > - local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > - printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > -
> > - "$@"
> > - local ret=$?
> > - if [ $ret -eq 0 ]; then
> > - echo "[PASS]"
> > - elif [ $ret -eq $ksft_skip ]; then
> > - echo "[SKIP]"
> > - exitcode=$ksft_skip
> > - else
> > - echo "[FAIL]"
> > - exitcode=1
> > - fi
> > + if test_selected ${CATEGORY}; then
> > + local title="running $*"
> > + local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > + printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > +
> > + "$@"
> > + local ret=$?
> > + if [ $ret -eq 0 ]; then
> > + echo "[PASS]"
> > + elif [ $ret -eq $ksft_skip ]; then
> > + echo "[SKIP]"
> > + exitcode=$ksft_skip
> > + else
> > + echo "[FAIL]"
> > + exitcode=1
> > + fi
> > + fi # test_selected
> > }
> >
> > -mkdir "$mnt"
> > -mount -t hugetlbfs none "$mnt"
> > +# setup only needed for hugetlb tests
> > +if test_selected "hugetlb"; then
> > + mkdir "$mnt"
> > + mount -t hugetlbfs none "$mnt"
> > +fi
> >
> > -run_test ./hugepage-mmap
> > +CATEGORY="hugetlb" run_test ./hugepage-mmap
> >
> > shmmax=$(cat /proc/sys/kernel/shmmax)
> > shmall=$(cat /proc/sys/kernel/shmall)
> > echo 268435456 > /proc/sys/kernel/shmmax
> > echo 4194304 > /proc/sys/kernel/shmall
> > -run_test ./hugepage-shm
> > +CATEGORY="hugetlb" run_test ./hugepage-shm
> > echo "$shmmax" > /proc/sys/kernel/shmmax
> > echo "$shmall" > /proc/sys/kernel/shmall
> >
> > -run_test ./map_hugetlb
> > +CATEGORY="hugetlb" run_test ./map_hugetlb
> >
> > -run_test ./hugepage-mremap "$mnt"/huge_mremap
> > -rm -f "$mnt"/huge_mremap
> > +CATEGORY="hugetlb" run_test ./hugepage-mremap "$mnt"/huge_mremap
> > +test_selected "hugetlb" && rm -f "$mnt"/huge_mremap
> >
> > -run_test ./hugepage-vmemmap
> > +CATEGORY="hugetlb" run_test ./hugepage-vmemmap
> >
> > -run_test ./hugetlb-madvise "$mnt"/madvise-test
> > -rm -f "$mnt"/madvise-test
> > +CATEGORY="hugetlb" run_test ./hugetlb-madvise "$mnt"/madvise-test
> > +test_selected "hugetlb" && rm -f "$mnt"/madvise-test
> >
> > -echo "NOTE: The above hugetlb tests provide minimal coverage. Use"
> > -echo " https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > -echo " hugetlb regression testing."
> > +if test_selected "hugetlb"; then
> > + echo "NOTE: These hugetlb tests provide minimal coverage. Use"
> > + echo " https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > + echo " hugetlb regression testing."
> > +fi
> >
> > -run_test ./map_fixed_noreplace
> > +CATEGORY="mmap" run_test ./map_fixed_noreplace
> >
> > # get_user_pages_fast() benchmark
> > -run_test ./gup_test -u
> > +CATEGORY="gup_test" run_test ./gup_test -u
> > # pin_user_pages_fast() benchmark
> > -run_test ./gup_test -a
> > +CATEGORY="gup_test" run_test ./gup_test -a
> > # Dump pages 0, 19, and 4096, using pin_user_pages:
> > -run_test ./gup_test -ct -F 0x1 0 19 0x1000
> > +CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
> >
> > -run_test ./userfaultfd anon 20 16
> > -run_test ./userfaultfd anon:dev 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd anon 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd anon:dev 20 16
> > # Hugetlb tests require source and destination huge pages. Pass in half the
> > # size ($half_ufd_size_MB), which is used for *each*.
> > -run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > -run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > -run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > rm -f "$mnt"/uffd-test
> > -run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > rm -f "$mnt"/uffd-test
> > -run_test ./userfaultfd shmem 20 16
> > -run_test ./userfaultfd shmem:dev 20 16
> > -
> > -#cleanup
> > -umount "$mnt"
> > -rm -rf "$mnt"
> > -echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem:dev 20 16
> > +
> > +# cleanup (only needed when running hugetlb tests)
> > +if test_selected "hugetlb"; then
> > + umount "$mnt"
> > + rm -rf "$mnt"
> > + echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > +fi
> >
> > -run_test ./compaction_test
> > +CATEGORY="compaction" run_test ./compaction_test
> >
> > -run_test sudo -u nobody ./on-fault-limit
> > +CATEGORY="mlock" run_test sudo -u nobody ./on-fault-limit
> >
> > -run_test ./map_populate
> > +CATEGORY="mmap" run_test ./map_populate
> >
> > -run_test ./mlock-random-test
> > +CATEGORY="mlock" run_test ./mlock-random-test
> >
> > -run_test ./mlock2-tests
> > +CATEGORY="mlock" run_test ./mlock2-tests
> >
> > -run_test ./mrelease_test
> > +CATEGORY="process_mrelease" run_test ./mrelease_test
> >
> > -run_test ./mremap_test
> > +CATEGORY="mremap" run_test ./mremap_test
> >
> > -run_test ./thuge-gen
> > +CATEGORY="hugetlb" run_test ./thuge-gen
> >
> > if [ $VADDR64 -ne 0 ]; then
> > - run_test ./virtual_address_range
> > + CATEGORY="hugevm" run_test ./virtual_address_range
> >
> > # virtual address 128TB switch test
> > - run_test ./va_128TBswitch.sh
> > + CATEGORY="hugevm" run_test ./va_128TBswitch.sh
> > fi # VADDR64
> >
> > # vmalloc stability smoke test
> > -run_test ./test_vmalloc.sh smoke
> > +CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
> >
> > -run_test ./mremap_dontunmap
> > +CATEGORY="mremap" run_test ./mremap_dontunmap
> >
> > -run_test ./test_hmm.sh smoke
> > +CATEGORY="hmm" run_test ./test_hmm.sh smoke
> >
> > # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
> > -run_test ./madv_populate
> > +CATEGORY="madv_populate" run_test ./madv_populate
> >
> > -run_test ./memfd_secret
> > +CATEGORY="memfd_secret" run_test ./memfd_secret
> >
> > # KSM MADV_MERGEABLE test with 10 identical pages
> > -run_test ./ksm_tests -M -p 10
> > +CATEGORY="ksm" run_test ./ksm_tests -M -p 10
> > # KSM unmerge test
> > -run_test ./ksm_tests -U
> > +CATEGORY="ksm" run_test ./ksm_tests -U
> > # KSM test with 10 zero pages and use_zero_pages = 0
> > -run_test ./ksm_tests -Z -p 10 -z 0
> > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 0
> > # KSM test with 10 zero pages and use_zero_pages = 1
> > -run_test ./ksm_tests -Z -p 10 -z 1
> > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 1
> > # KSM test with 2 NUMA nodes and merge_across_nodes = 1
> > -run_test ./ksm_tests -N -m 1
> > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 1
> > # KSM test with 2 NUMA nodes and merge_across_nodes = 0
> > -run_test ./ksm_tests -N -m 0
> > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
> >
> > # protection_keys tests
> > if [ -x ./protection_keys_32 ]
> > then
> > - run_test ./protection_keys_32
> > + CATEGORY="pkey" run_test ./protection_keys_32
> > fi
> >
> > if [ -x ./protection_keys_64 ]
> > then
> > - run_test ./protection_keys_64
> > + CATEGORY="pkey" run_test ./protection_keys_64
> > fi
> >
> > exit $exitcode
> > --
> > 2.31.1
> >
>
