Dear LO developers,
I already interviewed Maarten, yesterday, who wrote some years ago scripts to run the coverage tests on LO, but they seem not to be executed for a long time:
A simple/naive idea:
Loading ODT files into LO while doing a coverage test.
Loading ODT files into LO while doing a coverage test.
The source code coverage DIFF will reveal the source code being used exclusively for the ODT feature DIFF of the two documents.
For example:
For example:
- Load a very simple test document, like a single paragraph with "Hello World" with LibreOffice and during load doing a code coverage test to identify all the source code, which is being used for loading the ODF test document.
- Enhance the simple test document by a single feature, like making the text bold and do the loading & code coverage again.
- The difference (DIFF) between the two source code coverages represents source code exclusively relevant to the feature bold.
- Future commits changing source code of such an "ODF feature area" would trigger the load/save roundtrip test of these special ODF test document(s).
The advantage is to have a relevant minimum regression test not taking long to execute.
Is there some LO build VM available that can be extended to reactivate the code coverage tests, so interested parties might follow this approach?
Especially, I am longing for a "black box script" where I can give a text file with a list of ODF files as an input parameter to be code-cover-tested and receive for each input ODF the code coverage as a text file. Is there any script that can e used?
In post-processing, the source code DIFF would than be created afterwards. Shouldn't be too difficult (famous last words)... ;-)
Is anyone interested to join this adventure? :-)
Cheers,
Svante
---------- Forwarded message ---------
Von: Maarten Hoes <hoes.maarten@xxxxxxxxx>
Date: Mo., 2. Nov. 2020 um 22:35 Uhr
Subject: Re: Code Coverage LibreOffice
To: Svante Schubert <svante.schubert@xxxxxxxxx>
Cc: Michael Stahl <mst@xxxxxxxxxxxxxxx>
Von: Maarten Hoes <hoes.maarten@xxxxxxxxx>
Date: Mo., 2. Nov. 2020 um 22:35 Uhr
Subject: Re: Code Coverage LibreOffice
To: Svante Schubert <svante.schubert@xxxxxxxxx>
Cc: Michael Stahl <mst@xxxxxxxxxxxxxxx>
Hi Svante/Michael,
I'm glad to be of help, if I can be. :)
First of all, a disclaimer: I'm not a developer. I can write Unix/Linux shell scripts, and used to be a professional Unix/Linux system administrator, but I couldn't write a 'hello world' program in C or C++ if my life depended on it.
Perhaps I should start with some basics. When talking about 'code coverage', people generally mean 'measure what source code gets executed' (generally by the execution of a test suite), so that you can see how much and what parts of the source code gets covered by the test suite.
There is a wiki page [1] that describes how to set up lcov/gcov on Linux for the LibreOffice codebase, but I can't tell how accurate or up to date that wiki page currently is. Years ago, I wrote a shell script to automate that [2] process. If I recall correctly, it used to be run automatically (daily ?) by Jenkins, but the host/vm the job ran on became unavailable ages ago, and no-one seemed to notice or care enough to restore it. Old examples of what the generated reports look like can be found on the dev-builds.libreoffice.org website [3].
The process basically looks somewhat like this:
- Compile your code using additional flags, so that a record will be kept of what code gets executed.
- Run the test suite (or whatever else of which you want to see what source code gets executed by running it).
- Generate a human readable report that shows what sourcecode got executed.
Now, on to your specific questions:
1. Yes, lcov/gcov code coverage will show you all the source code lines that get executed by loading your 'Hello World' test document.
2. Yes, loading the same document multiple times will execute the exact same source code every time you do it, provided you don't change any of the conditions (like changing the source code in between runs).
3. Yes, I believe that if you take your 'Hello World' test document, and make that text bold, then the difference between the two runs should be the additional code that gets executed by making the text bold.
One thing to note though, as it may not be immediately obvious: When you start LibreOffice, and open a document, a lot more code will get executed than 'just' the code needed to display the text in the document. All of this additional code will get marked as having been executed in the report. I know of no way around that. So I am assUming (but you know what happens when we assUme) that you need to be intimately familiar with the entire LibreOffice codebase in order to differentiate between 'this is code that gets executed on startup, this is code needed for the GUI, this is code needed for parsing the document' and 'this is the code for the bold feature I want to test'. So I'm not so sure if this gets you specifically what you are looking for. (PS: There is a way to filter out / omit source code that you are not interested in in the final report, but this filtering is done on a per directory-basis. So you can for example filter out 'src/but/not/this/directory', but then you need to be very sure up-front that the code you are looking for is not contained in that directory. This also won't help you if you have 'code you are interested in' and 'code not interested in' that are both located in the same directory).
I hope this may assist to get you started. As I may have raised more questions than I attempted to answer, please feel free to let me know, and I will try my best to be of further assistance.
- Maarten
[1]
https://wiki.documentfoundation.org/Development/Lcov
[2]
https://git.libreoffice.org/buildbot/+/refs/heads/master/lcov-report/
[3]
I'm glad to be of help, if I can be. :)
First of all, a disclaimer: I'm not a developer. I can write Unix/Linux shell scripts, and used to be a professional Unix/Linux system administrator, but I couldn't write a 'hello world' program in C or C++ if my life depended on it.
Perhaps I should start with some basics. When talking about 'code coverage', people generally mean 'measure what source code gets executed' (generally by the execution of a test suite), so that you can see how much and what parts of the source code gets covered by the test suite.
There is a wiki page [1] that describes how to set up lcov/gcov on Linux for the LibreOffice codebase, but I can't tell how accurate or up to date that wiki page currently is. Years ago, I wrote a shell script to automate that [2] process. If I recall correctly, it used to be run automatically (daily ?) by Jenkins, but the host/vm the job ran on became unavailable ages ago, and no-one seemed to notice or care enough to restore it. Old examples of what the generated reports look like can be found on the dev-builds.libreoffice.org website [3].
The process basically looks somewhat like this:
- Compile your code using additional flags, so that a record will be kept of what code gets executed.
- Run the test suite (or whatever else of which you want to see what source code gets executed by running it).
- Generate a human readable report that shows what sourcecode got executed.
Now, on to your specific questions:
1. Yes, lcov/gcov code coverage will show you all the source code lines that get executed by loading your 'Hello World' test document.
2. Yes, loading the same document multiple times will execute the exact same source code every time you do it, provided you don't change any of the conditions (like changing the source code in between runs).
3. Yes, I believe that if you take your 'Hello World' test document, and make that text bold, then the difference between the two runs should be the additional code that gets executed by making the text bold.
One thing to note though, as it may not be immediately obvious: When you start LibreOffice, and open a document, a lot more code will get executed than 'just' the code needed to display the text in the document. All of this additional code will get marked as having been executed in the report. I know of no way around that. So I am assUming (but you know what happens when we assUme) that you need to be intimately familiar with the entire LibreOffice codebase in order to differentiate between 'this is code that gets executed on startup, this is code needed for the GUI, this is code needed for parsing the document' and 'this is the code for the bold feature I want to test'. So I'm not so sure if this gets you specifically what you are looking for. (PS: There is a way to filter out / omit source code that you are not interested in in the final report, but this filtering is done on a per directory-basis. So you can for example filter out 'src/but/not/this/directory', but then you need to be very sure up-front that the code you are looking for is not contained in that directory. This also won't help you if you have 'code you are interested in' and 'code not interested in' that are both located in the same directory).
I hope this may assist to get you started. As I may have raised more questions than I attempted to answer, please feel free to let me know, and I will try my best to be of further assistance.
- Maarten
[1]
https://wiki.documentfoundation.org/Development/Lcov
[2]
https://git.libreoffice.org/buildbot/+/refs/heads/master/lcov-report/
[3]
On Mon, Nov 2, 2020 at 8:55 PM Svante Schubert <svante.schubert@xxxxxxxxx> wrote:
Hi Maarten,Michael Stahl - who is as me a TDF member - dropped your name after the OASIS ODF TC call, in which we are both participating.I was curious about LO Code Coverage, but I am more working on ODF than on LibreOffice itself, for instance, the https://github.com/tdf/odftoolkitThere is a little experiment I have in mind, my hope is that you might get curious enough to join it ;-)The simple idea:
- Load a very simple test document, like a single paragraph with "Hello World" with LibreOffice and during load doing a code coverage of all the source code being used for loading the test document.
- Load the simple test document multiple times, is it always the same code coverage? (As I said I am new to LO Code and do not know the result)
- If it is the same (or changes within a predictable range), enhance the simple test document by a single feature, like making the text bold and do the loading & code coverage again. Is the difference between the two coverages the feature bold?
The question is: Is it possible to create an ODF feature <=> source code relation by doing the above?The advantage, if certain parts of source code are being changed, we have better insights on the ODF Testdocuments we are in need.The reason why I am asking is that the ODF Toolkit has a library ODFDOM, which is able to transform an ODT document into an equivalent list of user changes (in JSON) as if the document was created from top to bottom. By this, we are able to define more precisely the feature delta of the coverage scenario.In the upcoming year, I am trying to do several things:
- Add to the OASIS ODF specification these user changes (you may call them as well "ODF user features" or "ODF user semantics" or simply "ODF feature")
In addition, to add a semantic dictionary.- Refactor the work on the ODFDOM of ODF Toolkit...(many details)...
I would be very curious to set-up an environment, which I can use as a black box, providing a document as input and receive the code coverage as output!🤗Would you be so kind to assist me in that, Maarten? As I have frankly little idea where to start ;-)Best regards,Svante
_______________________________________________ LibreOffice mailing list LibreOffice@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/libreoffice
