On Fri, 7 Feb 2020, Masami Hiramatsu wrote:
> On Fri, 7 Feb 2020 08:27:13 +0000 (GMT)
> Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
>
> > On Fri, 7 Feb 2020, Masami Hiramatsu wrote:
> >
> > > Hi Alan,
> > >
> > > On Thu, 6 Feb 2020 15:09:20 +0000
> > > Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
> > >
> > > > In a number of cases, the ftrace tests check for the presence of
> > > > ftrace testing-related modules (ftrace-direct, trace-printk) and
> > > > programs (checkbashisms), returning exit_unresolved if these
> > > > are not found. The problem is, exit_unresolved causes execution
> > > > of ftracetest to return an error, when really our tests are
> > > > failing due to not having the requisite kernel configuration/tools
> > > > present, which is I think more of an unsupported error condition.
> > > > With these fixed, we see no unresolved test cases and ftracetest
> > > > returns success ("ok" when run via kselftest).
> > >
> > > If your problem is to pass the test even if you don't test the
> > > feature, please change the ftracetest itself instead of replacing
> > > unresolved with unsupported. Those notice different situation.
> > >
> > > unresolved - Testcase can not find some tools or helper drivers
> > > which are required for this testcase.
> > >
> > > unsupported - Kernel does not have tested feature because of
> > > the version or the configuration.
> > >
> > > Obviously the unresolved is a test environment issue. No test-module
> > > doesn't mean no feature to be tested.
> > > Could you tell me the reason why you can't install those required
> > > tools and modules on the test environment?
> > >
> >
> > Sure! In my case, I'm testing a distro production kernel,
> > where I can't control the CONFIG variable settings. In
> > this case, ideally I'd like the tests to return success
> > if no problems with ftrace were detected, even if some
> > of the tests could not be run due to missing modules
> > and programs.
>
> OK, for modules, we need to find another way to solve the issue.
> But how about checkbashisms? you can download and build it.
>
> https://sources.debian.org/src/devscripts/2.20.2/
>
Yep, I should have said that this one isn't a big issue, it's
also packaged in some distros in rpmdevtools.
> For the modules, you might be able to build it from kernel
> source code as out-of-tree modules, or not?
> (hmm, how do the other test handle it...?)
>
Ideally (from my perspective at least) the tests would
build the modules if needed, but I think the constraints
on kselftests are that sometimes the kselftests are packaged
and installed without the rest of the source tree, so I
_think_ given that the module source is in other parts of the
tree (that may not be present) we're probably stuck. It'd be
great to have have a solution to this though, as I have a
feeling there may be more and more cases like this with the
growth of kunit.
Looks like the official way to do this sort of thing is
described in Documentation/dev-tools/kselftest.rst:
# Assumes you have booted a fresh build of this kernel tree
cd /path/to/linux/tree
make kselftest-merge
make modules
sudo make modules_install
make TARGETS=lib kselftest
It seems to merge existing config with the config files in the
kselftest dirs and rebuild modules; I'll give this a try.
I was hoping for a lightweight version of the above which
just builds the modules needed without rebuilding everything.
> > As you suggest above (unless I'm
> > misunderstanding), this could be accomplished by modifying
> > ftracetest itself. Would doing something like what is done
> > for UNSUPPORTED_RESULT (defaults to 0, but can be set to
> > 1 via --fail-unsupported, such that ftracetest returns
> > 1 if we encounter unsupported results) make sense for
> > the unresolved case too?
>
> Yes, but at first could you try to setup your testing environment?
> If you are officially testing your distro kernel, the distro
> might need to be tested with full-set of testcases.
>
Absolutely! I'll see if I can convince the modules to
build using the above scheme.
Thanks for the review and advice!
Alan
> If not (like you are testing kernel for fun :)), you can just
> make your custom set of testcases. (just remove those test files)
>
> Thank you,
>
>
> --
> Masami Hiramatsu <mhiramat@xxxxxxxxxx>
>
