Tiago Botelho <tiagonbotelho@xxxxxxxxx> writes: >> Running this test number of times gives me spurious errors. Is the >> order of these output lines unstable? How do we "sort" these >> bisect-all results? If we are not sorting and the output depends on >> happenstance, then probably we would need to compare the expected >> and actual output after sorting. Or if the output depends on >> something under our control (e.g. they are related to topology and >> relative commit timestamp), we probably should try to control that >> "something" tighter so that we can rely on the order of the lines in >> the "expect" file. > > The reason why the tests were failing was because the above "old" tests > did not make use of test_commit which in turn would make the sha of each > commit be different and as a result give unexpected outputs at times. > If I move them to the top of that file the tests will pass every time, > would that > be ok? That means if somebody else needs to add other tests before the location you are planning to move your tests, your tests would suddenly start failing, no? That does not sound like a particularly robust solution to me.
