On Tue, 2020-11-17 at 01:34 +0300, Roman Bolshakov wrote: > On Fri, Nov 13, 2020 at 04:58:36PM +0100, Michal Privoznik wrote: > > On 11/8/20 10:24 PM, Roman Bolshakov wrote: > > > + if data['name'] == 'qemuxml2argvtest' and host_machine.system() == 'darwin' > > > + timeout = 180 > > > + else > > > + # default meson timeout > > > + timeout = 30 > > > + endif > > > + test(data['name'], test_bin, env: tests_env, timeout: timeout) > > > endforeach > > > > I think the last time I wanted to increase the timeout I was told that it is > > machine specific and since I know my machine the best I should use: meson > > test --timeout-multiplier=X > > But qumexml2argvtest has 1000+ test cases. And prior to the series the > test never fully passed. In some sense the series raises a baseline (but > with a higher timeout than on Linux). > > > Do you need this for the next cirrus patch? If so I think we should just add > > --timeout-multiplier= there. > > No, I needed it for local environment. And I profiled it a bit to find > where the time is spent [1]. I can also build a flamegraph if it helps. > > FWIW, it takes less time in Cirrus CI [2] than on MBP Pro 2012 but > timeout margin is quite narrow for default timeout: > > 90/127 qemuxml2argvtest OK 29.4992618560791 s > > So, I've measured test execution time on a few apple laptops at hand. > > On 16" MBP 2019 I get: > 90/285 libvirt / qemuxml2argvtest OK 22.11s > > On 15" MBP 2012: > 80/117 qemuxml2argvtest OK 48.31s > > On 13" MBA 2015 running on battery: > 80/117 qemuxml2argvtest OK 50.20s > > On 13" MBA 2015 running on A/C power: > 80/117 qemuxml2argvtest OK 48.71s > > Without the explicit timeout everyone has to remember that at least "-t 2" > has to be always passed to "meson test" on darwin. In my opinion, > project defaults should be reasonable without tweaking, to allow nice > "meson test -C build" to run all tests and be sure that they run without > failures if no issues are actually introduced, i.e. there should be no > false positives. This is especially important for "meson dist". > > What do you think about other timeout values for qemuxml2argvtest on > darwin: 60s, 90s, 120s, 150s? Is there a chance that one of them would > be ok? It's true that we make accomodations so that things work out of the box as much as possible, and this would fall in line with that trend. You've almost convinced we want this, but yeah we should make this maybe 90s - 180s seems quite excessive, given your measurements. I wonder if we should just bump the timeout for this test unconditionally: we know it's generally the one that takes the longest, so even on non-Apple hardware we might very well find that all other tests finish before the default timeout kicks in but this one can't quite make it. That's pretty much hwhat happened not too long ago in Debian: then 6.7.0 was imported, it FTBFS on a bunch of slow architectures due to this specific test timing out: https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=armel&ver=6.7.0-1&stamp=1599582160&raw=0 The solution was to pass --timeout-multiplier https://salsa.debian.org/libvirt-team/libvirt/-/commit/b327f9a356c81657aef905eb0d94f188d7a805bd https://salsa.debian.org/libvirt-team/libvirt/-/commit/2a7b4f4ad9285bfc9e6dd2d248e426924ef01140 Even if you bumped the timeout unconditionally, we wouldn't be able to drop that patch: if you look at the latest build for mipsel https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=mipsel&ver=6.9.0-1&stamp=1605523316&raw=0 for example, you'll see that *several* tests take longer than 30s to finish, even those that take a fraction of qemuxml2argv on x86_64, so I wouldn't be too keen on keeping track of them and singling them out in meson.build. But, one thing is old, slow architectures that people almost certainly don't do day-to-day development on, and another thing is relatively recent MacBooks :) -- Andrea Bolognani / Red Hat / Virtualization
