[RFC/PATCH v2] Better control of the tests run by a test suite

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is an update verson of the patches I've posted here:

    [RFC/PATCH] Better control of the tests run by a test suite
    http://www.mail-archive.com/git@xxxxxxxxxxxxxxx/msg46419.html

Chanes are only in the first patch, according to

    http://www.mail-archive.com/git@xxxxxxxxxxxxxxx/msg46423.html
    Ramsay Jones

and

    http://www.mail-archive.com/git@xxxxxxxxxxxxxxx/msg46512.html
    Eric Sunshine

The description below is identical to the previous one, but here
it is for convenience if someone would want to quote it in a
comment.

---

Hello,

This is a second attempt on a functionality I proposed in

    [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests
    http://www.mail-archive.com/git%40vger.kernel.org/msg44828.html

except that the implementation is quite different now.

I hope that I have accounted for the comments that were voiced so
far.  Let's see 

The idea behind the change is that sometimes it is convenient to
run only certain tests from a test suite.  Specifically I am
thinking about the following two use cases:

 1. You are developing new functionality.  You add a test that
    fails and then you add and modify code to make it pass.
    
 2. You have a failed test and you need to understand what is
    wrong.
    
In the first case you when you run the test suite, you probably
want to run some setup tests and then only one test that you are
focused on.

For code written in C time between you make a change and see a
test result is considerably increased by the compilation.  But
for script code turn around time is mostly due to the run time of
the test suite itself. [1]

For the second case you actually want the test suite to stop
after the failing test, so that you can examine the trash
directory without any modifications from the subsequent tests.
You probably do not care about them.

In the previous patch I've added an environment variable to
control tests to be run in a test suite.  I thought that it would
be similar to an already existing GIT_SKIP_TESTS.  As I did not
provide a cover letter that caused some misunderstanding I think.

This patch adds new command line argument '--run' that accepts a
list of patterns and restrictions on the test numbers that would
be included or excluded from this run of the test suite.

During discussion of the previous patch there were some talks
about extending GIT_SKIP_TESTS syntax.  In particular here:

> That is
>
>         GIT_SKIP_TESTS='t9??? !t91??'
>
> would skip nine-thousand series, but would run 91xx series, and all
> the others are not excluded.
>
> Simple rules to consider:
>
>  - If the list consists of _only_ negated patterns, pretend that
>    there is "unless otherwise specified with negatives, skip all
>    tests", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you
>    would treat GIT_SKIP_TESTS='* !t91??'.
>
>  - The orders should not matter for simplicity of the semantics;
>    before running each test, check if it matches any negative (and
>    run it if it matches, without looking at any positives), and
>    otherwise check if it matches any positive (and skip it if it
>    does not).
>
> Hmm?

    http://www.mail-archive.com/git%40vger.kernel.org/msg44922.html


I've used that as a basis, but the end result is somewhat
different.  Plus I've added boundary checks as in "<123".

Here are some examples of how functionality added by the patch
could be used.  In order to run setup tests and then only a
specific test (use case 1) one can do:

    $ ./t0000-init.sh --run='1 2 25'

or:

    $ ./t0000-init.sh --run='<3 25'

('<=' is also supported, as well as '>' and '>=').

In order to run up to a specific test (use case 2) one can do:

    $ ./t0000-init.sh --run='<=34'

or:

    $ ./t0000-init.sh --run='!>34'

Simple semantics described above is easy to implement, but at the
same time is probably not that convenient.  Rules implemented by
the patch are as follows:

 - Order does matter.  Whatever is specified on the right has
   higher precedence.

 - When the first pattern is positive the initial set of the
   tests to be run is empty.  You are adding to an empty set as
   in '1 2 25'.

   When the first pattern is negative the initial set of the
   tests to run contains all the tests.  You are subtracting
   from that set as in '!>34'.

It seems that for simple cases that gives simple syntax and is
almost unbiased if you think about preferring inclusion over
exclusion.  While it is unlikely it also allows for complicated
expressions.  And the implementation is quite simple as well.

Main functionality is in the third patch.  First two are just
minor fixes in related parts of the code.

P.S. I did not reply to the previous patch thread as this one is
quite different.


[1] Here are some times I see on Cygin:

    $ touch builtin/rev-parse.c
    
    $ time make
    ...
    
    real    0m10.382s
    user    0m3.829s
    sys     0m5.269s

Running the t0000-init.sh test suite is like this:

    $ time ./t0001-init.sh
    [...]
    1..36

    real    0m6.693s
    user    0m1.505s
    sys     0m3.937s

If I run only the 1, 2, 4 and 5th tests, it only half the time to
run the tests:

    $ time GIT_SKIP_TESTS='t0001.[36789] t0001.??' ./t0001-init.sh
    [...]
    1..36

    real    0m3.313s
    user    0m0.769s
    sys     0m1.844s 

Overall the change is from 17 to 14 seconds it is not that big.

If you only consider the test suite, as you do while you develop
an sh based tool, for example, the change is from 6.6 to 3.3
seconds.  That is quite noticeable.


 t/README         |   73 ++++++++++++--
 t/t0000-basic.sh |  296 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 t/test-lib.sh    |   96 +++++++++++++++++-
 3 files changed, 453 insertions(+), 12 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]