On Fri, Jun 21, 2019 at 12:21 PM shuah <shuah@xxxxxxxxxx> wrote: > > On 6/21/19 12:13 PM, Theodore Ts'o wrote: > > On Fri, Jun 21, 2019 at 08:59:48AM -0600, shuah wrote: > >>>> ### But wait! Doesn't kselftest support in kernel testing?! > >>>> > >>>> .... > >> > >> I think I commented on this before. I agree with the statement that > >> there is no overlap between Kselftest and KUnit. I would like see this > >> removed. Kselftest module support supports use-cases KUnit won't be able > >> to. I can build an kernel with Kselftest test modules and use it in the > >> filed to load and run tests if I need to debug a problem and get data > >> from a system. I can't do that with KUnit. Sure, I think this point has been brought up a number of times before. Maybe I didn't write this section well because, like Frank said, it comes across as being critical of the Kselftest module support; that wasn't my intention. I was speaking from the perspective that Kselftest module support is just a feature of Kselftest, and not a full framework like KUnit is (obviously Kselftest itself *is* a framework, but a very small part of it is not). It was obvious to me what Kselftest module support was intended for, and it is not intended to cover the use case that KUnit is targeting. > >> In my mind, I am not viewing this as which is better. Kselftest and > >> KUnit both have their place in the kernel development process. It isn't > >> productive and/or necessary to comparing Kselftest and KUnit without a > >> good understanding of the problem spaces for each of these. Again, I didn't mean to draw a comparison of which is better than the other. I was just trying to point out that Kselftest module support doesn't make sense as a stand alone unit testing framework, or really a framework of any kind, despite how it might actually be used. > >> I would strongly recommend not making reference to Kselftest and talk > >> about what KUnit offers. I can see your point. It seems that both you and Frank seem to think that I drew a comparison between Kselftest and KUnit, which was unintended. I probably should have spent more time editing this section, but I can see the point of drawing the comparison itself might invite this confusion. > > Shuah, > > > > Just to recall the history, this section of the FAQ was added to rebut > > the ***very*** strong statements that Frank made that there was > > overlap between Kselftest and Kunit, and that having too many ways for > > kernel developers to do the identical thing was harmful (he said it > > was too much of a burden on a kernel developer) --- and this was an > > argument for not including Kunit in the upstream kernel. I don't think he was actually advocating that we don't include KUnit, maybe playing devil's advocate; nevertheless, at the end, Frank seemed to agree that there were valuable things that KUnit offered. I thought he just wanted to make the point that I hadn't made the distinction sufficiently clear in the cover letter, and other reviewers might get confused in the future as well. Additionally, it does look like people were trying to use Kselftest module support to cover some things which really were trying to be unit tests. I know this isn't really intended - everything looks like a nail when you only have a hammer, which I think Frank was pointing out furthers the above confusion. In anycase, it sounds like I have, if anything, only made the discussion even more confusing by adding this section; sorry about that. > > If we're past that objection, then perhaps this section can be > > dropped, but there's a very good reason why it was there. I wouldn't > > Brendan to be accused of ignoring feedback from those who reviewed his > > patches. :-) > > > > Agreed. I understand that this FAQ probably was needed at one time and > Brendan added it to address the concerns. I don't want to speak for Frank, but I don't think it was an objection to KUnit itself, but rather an objection to not sufficiently addressing the point about how they differ. > I think at some point we do need to have a document that outlines when > to KUnit and when to use Kselftest modules. I think one concern people > have is that if KUnit is perceived as a replacement for Ksefltest > module, Kselftest module will be ignored leaving users without the > ability to build and run with Kselftest modules and load them on a need > basis to gather data on a systems that aren't dedicated strictly for > testing. I absolutely agree! I posed a suggestion here[1], which after I just now searched for a link, I realize for some reason it didn't seem like it reached a number of the mailing lists that I sent it to, so I should probably resend it. Anyway, a summary of what I suggested: We should start off by better organizing Documentation/dev-tools/ and create a landing page that groups the dev-tools by function according to what person is likely to use them and for what. Eventually and specifically for Kselftest and KUnit, I would like to have a testing guide for the kernel that explains what testing procedure should look like and what to use and when. > I am trying to move the conversation forward from KUnit vs. Kselftest > modules discussion to which problem areas each one addresses keeping > in mind that it is not about which is better. Kselftest and KUnit both > have their place in the kernel development process. We just have to be > clear on usage as we write tests for each. I think that is the right long term approach. I think a good place to start, like I suggested above, is cleaning up Documentation/dev-tools/, but I think that belongs in a (probably several) follow-up patchset. Frank, I believe your objection was mostly related to how KUnit is presented specifically in the cover letter, and doesn't necessarily deal with the intended use case. So I don't think that doing this, especially doing this later, really addresses your concern. I don't want to belabor the issue, but I would also rather not put words in your mouth, what are your thoughts on the above? I think my main concern moving forward on this point is that I am not sure that I can address the debate that this section covers in a way that is both sufficiently concise for a cover letter, but also doesn't invite more potential confusion. My inclination at this point is to drop it since I think the set of reviewers for this patchset has at this point become fixed, and it seems that it will likely cause more confusion rather than reduce it; also, I don't really think this will be an issue for end users, especially once we have proper documentation in place. Alternatively, I guess I could maybe address the point elsewhere and refer to it in the cover letter? Or maybe just put it at the end of the cover letter? [1] https://www.mail-archive.com/kgdb-bugreport@xxxxxxxxxxxxxxxxxxxxx/msg05059.html
