On 28.09.2023 22:54, Rae Moar wrote:
> On Mon, Sep 25, 2023 at 1:58 PM Michal Wajdeczko
> <michal.wajdeczko@xxxxxxxxx> wrote:
>>
>> In case of parameterized tests we are not providing a test plan
>> so we can't detect if any result is missing.
>>
>> Count available params using the same generator as during a test
>> execution
>>
>> Signed-off-by: Michal Wajdeczko <michal.wajdeczko@xxxxxxxxx>
>> Cc: David Gow <davidgow@xxxxxxxxxx>
>> Cc: Rae Moar <rmoar@xxxxxxxxxx>
>> ---
>> lib/kunit/test.c | 16 ++++++++++++++++
>> 1 file changed, 16 insertions(+)
>>
>> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
>> index 43c3efc286e4..55eabb324f39 100644
>> --- a/lib/kunit/test.c
>> +++ b/lib/kunit/test.c
>> @@ -540,6 +540,20 @@ static void kunit_accumulate_stats(struct kunit_result_stats *total,
>> total->total += add.total;
>> }
>>
>> +static size_t count_test_case_params(struct kunit_case *test_case)
>> +{
>> + char param_desc[KUNIT_PARAM_DESC_SIZE];
>> + const void *param_value = NULL;
>> + size_t num = 0;
>> +
>> + if (test_case->generate_params)
>> + while ((param_value = test_case->generate_params(param_value,
>> + param_desc)))
>> + num++;
>> +
>> + return num;
>> +}
>> +
>
> Hello!
>
> This change largely looks good to me. However, I am not 100 percent
> confident that the function to generate parameters always produces the
> same output (or same number of test cases). I would be interested in
> David's opinion on this.
Right, it's not explicitly specified in KUNIT_CASE_PARAM nor
test_case.generate_params documentation, but I would assume that while
generating different output could be fine (and harmless to this patch),
like based on a random seed, but IMO at the same time it should be
prohibited to generate different number of params, as this would make
harder to compare each execution for regression.
Alternatively we can introduce some flag to indicate whether provided
param generator is stable or not and then provide test plan only for the
former.
Michal
>
> Otherwise it seems to work well!
>
> Thanks!
> -Rae
>
>> int kunit_run_tests(struct kunit_suite *suite)
>> {
>> char param_desc[KUNIT_PARAM_DESC_SIZE];
>> @@ -585,6 +599,8 @@ int kunit_run_tests(struct kunit_suite *suite)
>> test_case->status = KUNIT_SKIPPED;
>> kunit_log_indent(KERN_INFO, &test, "KTAP version 1\n");
>> kunit_log_indent(KERN_INFO, &test, "# Subtest: %s", test_case->name);
>> + kunit_log_indent(KERN_INFO, &test, "1..%zd\n",
>> + count_test_case_params(test_case));
>>
>> while (test.param_value) {
>> kunit_run_case_catch_errors(suite, test_case, &test);
>> --
>> 2.25.1
>>
