> Please take this as stream of conscious feedback from a Fedora > contributor who is a casual maintainer of a few packages. I'm really > excited to see this working and not meaning to be critical; I hope this > is useful. > > Okay, so, I submitted an update, and — cool! — there are Automated Test > Results sitting there. And some of them have a subtle little ⓧ , which > I know means "failed" rather than "click to hide this" Agreed, the icons could be more obvious and visible. That would be an RFE against Bodhi. > because those > are highlighted in pink. Another one has a kind of mellon orange and a > circled !, so I guess it need my attention. > > So, click on the first failure. It's from rpmlint, and I know how to > read that output. Okay, this is easy — plain text with FAILED at the > bottom, with "(1 errors, 7 warnings)". I read upwards and see the > errors and warnings. Some of them are a little obscure, but I'm used to > rpmlint. > > Ok, next one — the orange warning from dist.rpmgrill. Woah — a whole > bunch of JSON. I search for "warning" or "notice" and get nothing; > there are bunch of "status" : "completed" lines. It's not at all clear > what I'm supposed to pay attention to. Hmmm. Yeah, rpmgrill output is hard to read (even I grind my teeth from time to time). Even though Fedora QA technically maintains the script wrapper (contributed by Ralph Bean), there's little to do there - rpmgrill doesn't seem to produce any output/artifact that would be more readable. This would need an RFE against rpmgrill. We as QA want to ideally execute as many tasks as our machines are able to, as long as they are at least mildly beneficial. The actual quality of those tasks and their outputs is going to differ wildly. Currently we directly maintain rpmlint, depcheck, upgradepath and dockerautotest. Rpmgrill and abicheck are maintained by external parties. We hope more of the latter will appear as taskotron gets more capable. We can try to devote some time to polishing external checks, but it's not going to scale. My current idea is that we will use task namespaces (currently we use mainly "dist.") to distinguish between "important" and "less important" tasks. We should strive to provide a good user experience for the important set, and will probably let the less important set live the life on its own. Bodhi can display just the important set, of both (with preference for the important one). > I notice that the next > result — the other failed one — is dist.rpmgrill.desktop-lint, so maybe > the warning is just telling me that a sub-test failed. Yes, it's just a sub-check result (and the log link goes to the full overall log, in case of rpmgrill). A few days ago I suggested how to show check results (collapsing sub-checks by default) here: https://github.com/fedora-infra/bodhi/issues/951#issuecomment-248843851 > There _is_ some > DesktopLint stuff in here, but I'm not quite clear on what I'm seeing > or what it means. There's stuff like "diag" : "Icon file > <var>ufraw</var> not found"... maybe that's it? > > Anyway, on to the failed apparent subtest. Oh look, more JSON. Hmmm. Is > this a subset of the previous? Just the relevant section? Looks like > it. Still don't know exactly what to do.... reading the JSON.... ah, > okay, it's not happy about the .desktop files that geeqie installs for > its own menus. That's a false positive; they're actually Fine For What > They Do. > > So, the one with a warning could definitely be more clear. For the > other JSON one, it really would be nice to have some explanation of > what's being tested, and have the JSON converted to something human > readable. I hope that's not just me being whiny — it feels barely > useable like this (especially as more new tests are addded). Yes, so far the effort was more on having them running (at all) than polish. We hope that some of that polish will come from the task maintainers. I would definitely not want to block-by-default on a task that is not well readable by a package maintainer (currently everything is just informational, the worst that can happen is karma autopush getting disabled - and you're informed about that). > > Beyond that... if I'm sure something is a false positive, is there a > way to mark it as suppressed? (I guess not hidden forever; maybe shown > at the bottom of the list as failed-but-okay'd or something.) I imagine > there are gonna be a lot of false positives. Not at the moment. Some tasks have specific override options (like with upstream rpmlint you can define you own config file with certain errors whitelisted - but we haven't implemented support for this in rpmlint task yet). We will need something universal, mainly for blocking-by-default checks (it could be as simple as allowing the maintainer to click "force push to stable" in bodhi and provide an explanation why he/she is ignoring the failed gating checks). However, none of this is implemented yet. Ignoring arbitrary (recurring) warnings from arbitrary tasks... might be a challenge. > > > Bonus question! Is there a UI for me to see these for packages which > don't go to bodhi? I usually just build into rawhide unless there is a > security update or major bugfix. Yes, resultsdb. From https://taskotron.fedoraproject.org/ your click on "Browse Task Results" and go to https://taskotron.fedoraproject.org/resultsdb/jobs . You can then use the search button to search for a specific NVR (please note that the other fields in the search button are somewhat broken, sorry), or write a custom query (double sorry), e.g. https://taskotron.fedoraproject.org/resultsdb/results?item=sudo-1.8.18-1.fc25 . You should also receive notifications about failed depcheck and upgradepath from FMN by default, and you can define your own rules to receive notifications about basically anything. See Martin's blogpost: https://mkrizek.wordpress.com/2016/02/24/taskotron-results-notifications/ I'd love to see a task results browser (or at least a link) in Koji UI, similar to what's currently in Bodhi (maybe it could even share some code?). Overall, thanks for your observations, and we should definitely do something about it. I tried to explain why it is the way it is (so much work and so little time, as usual) - currently it's more of a display of all that is currently being executed inside Taskotron rather than a well-thought package maintainer experience. Any RFE reported (against our tools or external tasks) is definitely very appreciated. Kamil _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
