On Tue, 2009-07-07 at 10:45 -0700, Martin Bligh wrote:
> On Tue, Jul 7, 2009 at 12:24 AM, sudhir kumar<smalikphy@xxxxxxxxx> wrote:
> > On Tue, Jul 7, 2009 at 12:07 AM, Martin Bligh<mbligh@xxxxxxxxxx> wrote:
> >>>> Issues: LTP has a history of some of the testcases getting broken.
> >>
> >> Right, that's always the concern with doing this.
> >>
> >>>> Anyways
> >>>> that has nothing to worry about with respect to autotest. One of the known issue
> >>>> is broken memory controller issue with latest kernels(cgroups and memory
> >>>> resource controller enabled kernels). The workaround for them I use is to
> >>>> disable or delete those tests from ltp source and tar it again with the same
> >>>> name. Though people might use different workarounds for it.
> >>
> >> OK, Can we encapsulate this into the wrapper though, rather than making
> >> people do it manually? in the existing ltp.patch or something?
> >>
> > definitely we can do that, but that needs to know about all the corner
> > cases of failure. So may be we can continue enhancing the patch as per
> > the failure reports on different OSes.
> >
> > 1 more thing I wanted to start a discussion on LTP mailing list is to
> > make aware the testcase if it is running on a physical host or on a
> > guest(say KVM guest). Testcases like power management, group
> > scheduling fairness etc do not make much sense to run on a guest(as
> > they will fail or break). So It is better for the test to recognise
> > the environment and not execute if it is under virtualization and it
> > is supposed to fail or break under that environment. Does that make
> > sense to you also ?
>
> Yup, we can pass an excluded test list. I really wish they'd fix their
> tests, but I've been saying that for 6 years now, and it hasn't happened
> yet ;-(
I would slightly disagree to that. 6 years is history. But, have you
recently checked with LTP ?
Yes, there were tests in LTP which were broken by design. And many of
them were fixed in last course of years. And probably this is the first
time in LTPś history when people fixed all build/broken issues in a
month. Recently people stopped complaining about the so-called broken
issues. But i am hearing it again. Could you please point to this
mailing list the issues that you still face. I am sure there are now
very active members who would help you fix them.
Few more observations. If the test cases design is broken, the reason
for it reporting BROKEN, then this is a genuine issues. We would need to
fix them. But, when a test case reports BROKEN, is it justified to put
the whole blame on the test case. Did we verify whether the issue can
also be with:
1. Libraries,
2. Kernel (the very purpose for which test case is designed),
What will be the point if we just want the test case to PASS ?? So, do
we want to point that the kernel is always OK, and we just want the test
cases to report PASS for it, else the test case is broken ?
Fixing build issues:
LTP cannot stop inheriting new test cases with each passing day. But,
how do we guarantee that the test suite build will succeed on:
1. every Distro,
2. every arch,
3. every kernel
Unlike the Linux Kernel, which carries all the stuff needed to build
itself on any arch, an user land project like LTP is completely
dependant on the system headers/libraries to complete itś build. So,
when these stuff are not consistent across the architecture and Distro
geography, so, how do we solve the problem. And in most of the instance
we find that 50% of our effort/code goes in fixing this dependencies,
rather than into the actual kernel test code. And, yes, to cater to the
need, LTP has embraced the AUTOCONF framework.
Fixing Real test case BROKEN issues:
Here plays one of the unfortunate drawbacks of the Open Source Projects.
Everything does not get funded for eternity. When test cases were
checked in for some feature in the kernel, the project should have been
funded. Now, when the ABIs/functionality in the kernel changes, the
original author of those test cases is no more funded to make the
corresponding changes in the concerned tests. So, this can be solved
only when somebody reports such issues with a patch rather than sending
repeated reminders that such-and-such test cases are broken. I do not
have any other idea of how this can be solved.
I am not sure, if i am able to resolve all your doubts/concerns
completely. Mike/others from LTP mailing list will may also be of some
help.
Mike,
What do you say ?
Regards--
Subrata
> _______________________________________________
> Autotest mailing list
> Autotest@xxxxxxxxxxxxxxx
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
