Hi Jeff, On 11/12/2014 11:21 PM, Jeff Darcy wrote:
I'm ambivalent. On the one hand, I think this is an important step in the right direction. Sometimes we need to be able to run *all* of our existing tests with some feature enabled, not just a few feature-specific tests. SSL is an example of this, and transport (or other forms of) multi-threading will be as well.
I agree, however this seems a lot more complex since the number of combinations grows exponentially. Maybe it should be discussed to create another script able to do so (or merge it with regressions).
The intended purpose of this modification is to verify that all volume types fulfill posix standards. Since most of the causes that can break posix compliance are located in the xlators that implement the different volume types, I though that smoke test (that currently executes the posix compliance test) was the right place.
On the other hand, I'm not sure smoke is the place to do this. Smoke is supposed to be a *quick* test to catch *basic* errors (e.g. source fails to build) before we devote hours to a full regression test. How much does this change throughput on the smoke-test queue? Should we be doing this in regression instead, or in a third testing tier between the two we have?
To try to minimize the effects of testing more configurations generating longer runs, I modified the script to test all volume types in parallel.
A simple tests seems to indicate that the test time does not increase much. However this would need to be tested on the jenkins' slaves to be sure.
On my test machine, a simple smoke test with a distributed-replicated volume took 155 seconds. A run with four volume types (distributed with one brick, distributed with three bricks, replicated and distributed-replicated) took 227 seconds. This represents an increase of 46%, while we have multiplied by 4 the number of tests.
I tested this on a quite slow machine, so it could be much better in jenkins' slaves.
My gut feel is that we need to think more about how to run a matrix of M tests across N configurations, instead of just putting feature/regression tests and configuration tests into one big bucket. Or maybe that's a longer-term thing.
I think that trying to parallelize regressions could reduce running time enormously, however this would need an important change in the testing infrastructure.
Xavi _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
