On Sat, Dec 5, 2015 at 11:49 AM, Loic Dachary <loic@xxxxxxxxxxx> wrote: > Hi Ceph, > > TL;DR: a ceph-qa-suite bot running on pull requests is sustainable and is an incentive for contributors to use teuthology-openstack independently A bot for scheduling a named suite on a named PR, and posting the results back the PR is definitely a good thing. Thinking further about using commit messages to toggle the testing, I think that this could get awkward when it's coupled to the human side of code review. When someone pushes a "how about this?" modification they don't necessarily want to re-run the test suite until the reviewer has okayed it, but then that means that they have to push again, and the final thing that's tested would be a different SHA1 (hopefully the same code) than what the human last reviewed. We'll also have e.g. rebases, where there tends to be some discretion about whether a rebase requires a re-test. When you were talking about having the suite selected in the qa: tag, there was the motivation to put it in the commit message so that it would be preserved in backports. However, if the "Needs-qa:" flag is just a boolean, then I think it makes more sense to control it with a github label or by posting a command in a PR comment. I'm not sure how this really helps with the resource issues; for example with the fs suite we would probably not be able to make a finer-grained choice about what tests to run based on the diff. The part about randomly dropping a subset of tests when resources are low doesn't make sense to me -- I think the bot should either give up or enqueue itself. Cheers, John > When a pull request is submitted, it is compiled, some tests are run[1] and the result is added to the pull request to confirm that it does not introduce a trivial problem. Such tests are however limited because they must: > > * run within a few minutes at most > * not require multiple machines > * not require root privileges > > More extensive tests (primarily integration tests) are needed before a contribution can be merged into Ceph [2], to verify it does not introduce a subtle regression. It would be ideal to run these integration tests on each pull request but there are two obstacles: > > * each test takes ~ 1.5 hour > * each test cost ~ 0.30 euros > > On the current master, running all tests would require ~1000 jobs [3]. That would cost ~ 300 euros on each pull request and take ~10 hours assuming 100 jobs can run in parallel. We could resolve that problem by: > > * maintaining a ceph-qa-suite map to be used as a white list mapping a diff to a set of tests. For instance, if the diff modifies the src/ceph-disk file, it outputs the ceph-disk suite[4]. This would effectively trim the tests that are unrelated to the contribution and reduce the number of tests to a maximum of ~100 [4] and most likely a dozen. > * tests are run if one of the commits of the pull request has the *Needs-qa: true* flag in the commit message[5] > * limiting the number of tests to fit in the allocated budget. If there was enough funding for 10,000 jobs during the previous period and there was a total of 1,000 test run required (a test run is a set of tests as produced by the ceph-qa-suite map), each run is trimmed to a maximum of ten tests, regardless. > > Here is an example: > > Joe submits a pull request to fix a bug in the librados API > The make check bot compiles and fails make check because it introduces a bug > Joe uses run-make-check.sh locally to repeat the failure, fixes it and repush > The make check bot compiles and passes make check > Joe amends the commit message to add *Needs-qa: true* and repushes > The ceph-qa-suite map script finds a change on the librados API and outputs smoke/basic/tasks/rados_api_tests.yaml > The ceph-qa-suite bot runs the test smoke/basic/tasks/rados_api_tests.yaml which fails > Joe examines the logs found at http://teuthology-logs.public.ceph.com/ and decides to debug by running the test himself > Joe runs teuthology-openstack --suite smoke/basic/tasks/rados_api_tests.yaml against his own OpenStack tenant [6] > Joe repush with a fix > The ceph-qa-suite bot runs the test smoke/basic/tasks/rados_api_tests.yaml which succeeds > Kefu reviews the pull request and has a link to the successful test runs in the comments > > This approach scales with the size of the Ceph developer community [7] because regular contributors benefit directly from funding the ceph-qa-suite bot. New contributors can focus on learning how to interpret the ceph-qa-suite error logs for their contribution and learn about how to debug it via teuthology-openstack if needed, which is a better user experience than trying to figure out which ceph-qa-suite job to run, learning about teuthology, schedule the test and interpret the results. > > The maintenance workload of a ceph-qa-suite bot probably requires one work day a week, to handle funding, sysadmin of the server where the bot runs but mostly to sort out the false negatives. I believe a pure self-service approach where each contributor would be asked to run teuthology-openstack independently would actually require more work. The ceph-qa-suite bot provides a baseline on which everybody can agree to sort out the false negatives. When a contributor runs teuthology-openstack by herself/himself, it is difficult for her/him to figure out if a failure comes from something she/he did incorrectly because she/he is not familiar with teuthology-openstack or if it is related to her/his contribution. She/He will asks for assistance in situations where comparing her/his run with the output of the ceph-qa-suite bot would probably give her/him enough hints to fix the problem herself/himself. > > If the ceph-qa-suite bot becomes unavailable, the contributors are not blocked because they can run it by themselves on their own OpenStack tenant and link the results to the pull request in the same way the bot would. Debugging a failed test is essentially the same thing as running the ceph-qa-suite bot. > > Cheers > > [1] run-make-check.sh https://github.com/ceph/ceph/blob/master/run-make-check.sh > [2] Ceph test suites https://github.com/ceph/ceph-qa-suite/tree/master/suites > [3] teuthology-suite --suite . --subset 1/40000 > [4] minimal number of tests to run all tasks at least once: 130 for rados, 76 for fs, 113 for upgrade, 18 for rgw, 45 for rbd. > [5] a former proposal was to include the test suite to run in the commit message, but this is more difficult to maintain that a boolean flag that states a given commit needs to pass all the relevant tests > [6] teuthology-openstack https://github.com/dachary/teuthology/tree/openstack#openstack-backend > [7] Scaling out the Ceph community lab http://dachary.org/?p=3852 > -- > Loïc Dachary, Artisan Logiciel Libre > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
