Re: [PATCH] selftests: Add support for argc and argv.

[Kuniyuki Iwashima <kuniyu@xxxxxxxxxxxx>]

From:   Kees Cook <keescook@xxxxxxxxxxxx>
Date:   Wed, 4 Mar 2020 10:17:41 -0800
> On Wed, Mar 04, 2020 at 05:52:04PM +0900, Kuniyuki Iwashima wrote:
> > Currently tests are often written in C and shell script. In many cases, the
> > script passes some arguments to the C program. However, the helper
> > functions do not support arguments, so many tests are written without
> > helper functions.
> > 
> > This patch allows us to handle argc and argv in each tests and makes it
> > easier to write tests flexibly with helper functions.
> > 
> > Signed-off-by: Kuniyuki Iwashima <kuniyu@xxxxxxxxxxxx>
> 
> Interesting! Do you have an example that uses this? I wonder if it might
> make sense instead to allow extending the struct __test_metadata with
> test-specific options so that individual tests don't have to re-parse
> argv every time (the main test running could instead do it once and set
> variables in struct __test_metadata (or somewhere else).

I added a sample test program at the end of this mail.

There are some functions that are not TEST() but are passed __test_metadata to
in order to use ASSERT_EQ in the function. I did not extend __test_metadata
because I thought argc and argv would not be used in such functions.

  e.g. kill_thread_or_group() in tools/testing/selftests/seccomp/seccomp_bpf.c

But, I have not thought about re-parsing, thank you!
Also I thought up that it is better to pass argc and argv to
FIXTURE_SETUP/TEARDOWN.

Now I have two idea.

  1. pass argc and argv to FIXTURE_SETUP/TEARDOWN.
  2. define COMMON_FIXTURE and COMMON_FIXTURE_SETUP/TEARDOWN,
       and pass COMMON_FIXTURE to all tests.
       (I think it is not good to extend __test_metadata because argc and
        argv is not metadata, so it is good to setup another vars with args)

I think each has pros and cons.

1.
Pros
  - shell script only has to call a C program once with some arguments and
      each FIXTURE_SETUP differs from one another
  - shell script can call the same C program with different arguments and
      each FIXTURE_SETUP differs from one another
Cons:
  - if TEST()s use the same FIXTURE, the same FIXTURE_SETUP is called in each TEST()s.

2.
Pros:
  - we do not have to re-parse argc and argv in each TEST()s.
Cons:
  - 1. may give more flexibility than 2.

Which would you think is better?
I would be happy if you tell me another idea!

Thanks.


===sample===
#include "./kselftest_harness.h"


TEST(argc_test)
{
	int i;
	for (i = 0; i < argc; i++)
		TH_LOG("argv[%d]: %s", i, argv[i]);
}

FIXTURE(argc_f) {
	int data;
};

FIXTURE_SETUP(argc_f) {
	self->data = 92;
	ASSERT_EQ(92, self->data);
}

FIXTURE_TEARDOWN(argc_f) {
}

TEST_F(argc_f, argc_test_f) {
	int i;
	for (i = 0; i < argc; i++)
		TH_LOG("fixture: %d\targv[%d]: %s", self->data, i, argv[i]);
}

TEST_HARNESS_MAIN
============