On 12/09/2018 10:07, Chris Wilson wrote:
Quoting Tvrtko Ursulin (2018-09-12 09:48:00)
On 07/09/2018 12:43, Chris Wilson wrote:
Quoting Tvrtko Ursulin (2018-09-07 12:14:20)
From: Tvrtko Ursulin <tvrtko.ursulin@xxxxxxxxx>
Proposal to add test tags as a replacement for separate test
list which can be difficult to maintain and get out of date.
Putting this maintanenace inline with tests makes it easier
to remember to update the (now implicit) lists, and also enables
richer test selection possibilities for the test runner.
Current method of implying tags from test/subtest names has a
couple of problems one of which is where some names can use
the same token for different meanings. (One example is the
"default" token.) It also creates a name clash between naming
and tagging.
Test tags are strings of tokens separated by spaces, meaning of
which we decide by creating a convetion and helped by the
suitable helper macros.
For example tags can be things like: gem, kms, basic, guc, flip,
semaphore, bz12345678, gt4e, reset, etc..
At runtime this would look something like this (abbreviated for
readability):
@ tests/gem_sync --list-subtests-with-tags
...
forked-each TAGS="gem "
forked-store-each TAGS="gem "
basic-all TAGS="gem basic "
basic-store-all TAGS="gem basic "
@ tests/gem_concurrent_blit --list-subtests-with-tags
...
16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "
@ tests/kms_flip --list-subtests-with-tags | grep basic
basic-plain-flip TAGS="kms basic "
basic-flip-vs-dpms TAGS="kms basic "
Test runner could then enable usages like:
./run-tests --include gem --exclude stress
Minor comment, I like some basic boolean algebra here
--include "gem AND smoketest NOT stress"
:)
That's what my hypothetical examples showed just with a different syntax!
I'd probably tag everything according to which ioctls they use. I feel my
endgoal is still to list tests by their kernel functions (which can and
should be automatically derived), and their hw function which is the
open problem.
If we want to do that automatically then tagging in this flavour
probably doesn't make sense and it should instead be an external database.
If we go on the ioctl granularity it can probably mostly have one
version, and it should be static enough to be able to live in the tree,
but if we go more granular, like something derived from kcov, then
that's out of the window. Since it will be tied both to a platform and
kernel version.
So I think tagging as proposed here is not appropriate for either, but
only if we wanted to tag on coarser functional areas and such.
Yeah, I think the same as well, that tags should be "smoke", "stress".
(But one man's stress is another's minimal workload :(
But both of those are in essence a duration parameter, and I'd prefer if
we expressed those as configurable parameters. At which point there is a
meta level of test scripts to tag ;)
I feel that "gem", "kms" is better expressed in lower level terms, so
what's left to tag? Clients? "display", "opencl", "media", "opengl" ?
Even using client specs for features (e.g. EGL_IMG_context_priority)?
If we overcomplicate and try to change too much at once it is deemed to
fail. GEM, KMS, etc, have this advantage of being very established.
So I was thinking these basic keywords and then something high level
like reset, rps, stolen, and similar. So for instance we can remove the
igt_skip_on_simulation, which can be time consuming to answer why it is
there, and replace it with --exclude reset,... when running under the
simulator.
True it is manual work to keep the tags up to date. Is it more work than
the current scheme is TBD.
Who and why would use those? From a selfish perspective, I want lcov
with tests sorted in order of increasing "stress" (start with the
precise X does Y test, finish with can it survive pathological client
behaviour for 48 hours).
Who and why would use which ones? Tags in general, tags in the style the
patch proposes, or tags in the flavour you described?
My initial idea was that it would make easy for developers to run an
approximate sweep when touching a feature. But with good CI and trybot
maybe no one would use tags and just rely on "outsourced" test runs?
Could be.. but hey, you convinced me to re-send this. Or was it Joonas? :)
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
Intel-gfx@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
