On 06/08/2017 01:38 PM, Shuah Khan wrote: > Hi Sumit, > > On 06/08/2017 11:46 AM, Sumit Semwal wrote: >> Hi Greg, >> >> On 8 June 2017 at 22:14, Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote: >>> On Thu, Jun 08, 2017 at 10:37:55AM -0500, Tom Gall wrote: >>>> We've been running kselftests for ARM and x86 hardware in an effort to >>>> detect regressions in various kernels including LTS and candidate >>>> patches. >>>> >>>> One general question we've had is what's the right thing to do when it >>>> comes to running kselftest on older LTS kernels. Let's pick on 4.4 for >>>> the purposes of this discussion tho obviously the example applies to >>>> other versions. >>>> >>>> 1) Run current top of tree, kselftest on a 4.4 LTS? >>>> 2) Run kselftest from 4.4 on 4.4 LTS? >>>> >>>> #1 gets latest greatest set of tests but obviously there can be >>>> breakage because of how the kernel evolves over time. >>> >>> Really? What breaks? >>> >> So in what we've observed so far, testing a 4.4 kernel with say 4.10 >> kselftests, most of the failures are due to features getting added >> later than the kernel being tested, but there are few that fail due to >> behaviour changes in features. > > That is the expected behavior. A few more thoughts and questions. New tests should detect features and not run tests and print a message indicating that the feature isn't supported or skip the test silently. When you say fail, does that mean skip running the test or marking the test as failed. It would be easier to for new feature to be skipped, however, it could be difficult to identify behavior changes in the feature. It depends what kind of behavior change. For example, adding a new flag to syscall, testing for new flag support would show up as a failure on an older kernel. This might be harder one for the test to detect just based on the feature presence. If you could send the results for such tests, we can chat and find a good way to fix it. thanks, -- Shuah