On 2022/12/16 00:42, David Disseldorp wrote:
Hi Qu,
On Thu, 15 Dec 2022 19:41:13 +0800, Qu Wenruo wrote:
Although the current result files "check.log" and "check.time" is enough
for human to read, it's not that easy to parse.
Have you looked at the existing junit XML based report types, available
via "check -R xunit ..."? junit is standardized, parsable and supported
by tools such as:
- https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
- https://github.com/weiwei/junitparser
- https://ddiss.github.io/online-junit-parser/
- https://plugins.jenkins.io/junit/
Oh, that's way better, the existing reporting facility is really what I
need.
Thus this patch will introduce a json output to "$RESULT_BASE/check.json".
The example output would look like this:
{
"section": "(none)",
"fstype": "btrfs",
"start_time": 1671103264,
"arch": "x86_64",
"kernel": "6.1.0-rc8-custom+",
"results": [
{
"testcase": "btrfs/001",
"status": "pass",
"start_time": 1671103264,
"end_time": 1671103266
},
{
"testcase": "btrfs/006",
"status": "pass",
"start_time": 1671103266,
"end_time": 1671103268
},
{
"testcase": "btrfs/007",
"status": "pass",
"start_time": 1671103268,
"end_time": 1671103271
}
]
}
Which should make later parsing much easier.
Signed-off-by: Qu Wenruo <wqu@xxxxxxxx>
---
Reason for RFC:
- Not crash safe
If one test case caused a crash, the "check.json" file will be an
invalid one, missing the closing "] }" string.
- Is json really a good choice?
It may be much easier to convert to a web page, but we will still
need to parse and handle the result using another languages anyway,
like to determine a regression.
I'm not opposed to adding an extra json report type, but I really think
it should be plumbed into the existing common/report API.
Another alternative is .csv, and it can be much easier to handle.
(pure "echo >> $output", no need to handle the comma rule).
But for .csv, we may waste a lot of columes for things like "arch",
"kernel", "section".
My preference for any new output formats, especially if they're intended
for parsing, is that they're based on an existing standard/tool. E.g.
https://testanything.org .
That's a much better unified protocol, I'll definitely look into it.
Thanks for all the feedbacks!
Qu
Cheers, David
