On Wednesday 08 July 2009 06:19:27 Subrata Modak wrote: > On Tue, 2009-07-07 at 10:45 -0700, Martin Bligh wrote: > > On Tue, Jul 7, 2009 at 12:24 AM, sudhir kumar<smalikphy@xxxxxxxxx> wrote: > > > On Tue, Jul 7, 2009 at 12:07 AM, Martin Bligh<mbligh@xxxxxxxxxx> wrote: > > >>>> Issues: LTP has a history of some of the testcases getting broken. > > >> > > >> Right, that's always the concern with doing this. > > >> > > >>>> Anyways > > >>>> that has nothing to worry about with respect to autotest. One of the > > >>>> known issue is broken memory controller issue with latest > > >>>> kernels(cgroups and memory resource controller enabled kernels). The > > >>>> workaround for them I use is to disable or delete those tests from > > >>>> ltp source and tar it again with the same name. Though people might > > >>>> use different workarounds for it. > > >> > > >> OK, Can we encapsulate this into the wrapper though, rather than > > >> making people do it manually? in the existing ltp.patch or something? > > > > > > definitely we can do that, but that needs to know about all the corner > > > cases of failure. So may be we can continue enhancing the patch as per > > > the failure reports on different OSes. > > > > > > 1 more thing I wanted to start a discussion on LTP mailing list is to > > > make aware the testcase if it is running on a physical host or on a > > > guest(say KVM guest). Testcases like power management, group > > > scheduling fairness etc do not make much sense to run on a guest(as > > > they will fail or break). So It is better for the test to recognise > > > the environment and not execute if it is under virtualization and it > > > is supposed to fail or break under that environment. Does that make > > > sense to you also ? > > > > Yup, we can pass an excluded test list. I really wish they'd fix their > > tests, but I've been saying that for 6 years now, and it hasn't happened > > yet ;-( > > I would slightly disagree to that. 6 years is history. But, have you > recently checked with LTP ? > > Yes, there were tests in LTP which were broken by design. And many of > them were fixed in last course of years. And probably this is the first > time in LTPś history when people fixed all build/broken issues in a > month. Recently people stopped complaining about the so-called broken > issues. But i am hearing it again. Could you please point to this > mailing list the issues that you still face. I am sure there are now > very active members who would help you fix them. > > Few more observations. If the test cases design is broken, the reason > for it reporting BROKEN, then this is a genuine issues. We would need to > fix them. But, when a test case reports BROKEN, is it justified to put > the whole blame on the test case. Did we verify whether the issue can > also be with: > > 1. Libraries, > 2. Kernel (the very purpose for which test case is designed), > > What will be the point if we just want the test case to PASS ?? So, do > we want to point that the kernel is always OK, and we just want the test > cases to report PASS for it, else the test case is broken ? > > Fixing build issues: > LTP cannot stop inheriting new test cases with each passing day. But, > how do we guarantee that the test suite build will succeed on: > 1. every Distro, > 2. every arch, > 3. every kernel > > Unlike the Linux Kernel, which carries all the stuff needed to build > itself on any arch, an user land project like LTP is completely > dependant on the system headers/libraries to complete itś build. So, > when these stuff are not consistent across the architecture and Distro > geography, so, how do we solve the problem. And in most of the instance > we find that 50% of our effort/code goes in fixing this dependencies, > rather than into the actual kernel test code. And, yes, to cater to the > need, LTP has embraced the AUTOCONF framework. > > Fixing Real test case BROKEN issues: > Here plays one of the unfortunate drawbacks of the Open Source Projects. > Everything does not get funded for eternity. When test cases were > checked in for some feature in the kernel, the project should have been > funded. Now, when the ABIs/functionality in the kernel changes, the > original author of those test cases is no more funded to make the > corresponding changes in the concerned tests. So, this can be solved > only when somebody reports such issues with a patch rather than sending > repeated reminders that such-and-such test cases are broken. I do not > have any other idea of how this can be solved. > > I am not sure, if i am able to resolve all your doubts/concerns > completely. Mike/others from LTP mailing list will may also be of some > help. not much else to say. it's an open source project and if you arent willing to contribute to fixing it but rather just sit back and complain, then you cant really be surprised when things dont move. -mike
Attachment:
signature.asc
Description: This is a digitally signed message part.
