John Spray writes: > On Wed, Apr 13, 2016 at 7:02 PM, Vasu Kulkarni <vakulkar@xxxxxxxxxx> wrote: >> >> >> On Wed, Apr 13, 2016 at 3:52 AM, John Spray <jspray@xxxxxxxxxx> wrote: >>> >>> Lately, we've instances where ceph.git and ceph-qa-suite.git got >>> slightly out of sync, as we were adding new stuff and interface >>> changes to ceph (especially in cephfs). >>> >>> We used to have two repos (ceph and teuthology), now we have three >>> (ceph, ceph-qa-suite and teuthology). >> >> >> we also have s3 suites in its own repo, we have big files in ceph.com/qa >> which i believe is the right place. > > I'm a bit ignorant of the s3 tests, is there a particular reason they > live separately to the main ceph-qa-suite? Technically s3-tests can have tests that do not all pass on rgw (which is true mostly even now as we have a fails_on_rgw filter) and pass on Amazon S3. I'm guessing this may be a reason they live outside of the qa-suite, but I could be wrong Abhishek > > As for the stuff in ceph.com/qa, I of course am not suggesting putting > those bit binaries into the main git repo. It is sort of annoying > that they live out there on a web server, but it's a separate issue > from the ceph/ceph-qa-suite one. > >>> Splitting tests out of >>> teuthology was a good thing, but maybe they should have gone into the >>> ceph tree instead of a new repo? The ceph-qa-suite branching seems to >>> pretty much mirror what we do with ceph, with master vs jewel etc. >>> >>> Historically we had a comparatively static set of workloads in the qa >>> suite (e.g. kernel-untar-build, fsstresss, pjd), which didn't change >>> much with ceph changes. But these days we're adding much more >>> detailed tests, so there's more effort to keep the two in sync. >>> >>> I would personally love to be able to have a single PR that contained >>> my code and the tests for it. What if after Jewel we pulled all of >>> ceph-qa-suite into the ceph repo? >> >> >> I understand the advantage of having same sha between builds and tests, but >> how are we going to guarantee ceph-builds in different places(one built on >> other than ceph.com) will have same sha? > > The thing that needs to match is the sha1 of the ceph code you're > running, and the sha1 you use when pulling the code to test it with. > > I believe you currently have a situation where you have a built RPM > but you're not sure what ceph sha1 it came from, so you're pulling > e.g. the jewel branch of ceph-qa-suite rather than a specific sha1. > That needs fixing independently of ceph/ceph-qa-suite separation: you > have to run the right revision of the tests, irrespective of where > they're getting pulled from. > > Basically, anyone generating packages with versions that don't map to > a ceph sha1 will need to have a way of remembering what ceph sha1 > their packages correspond to. This is not a new requirement, it's the > same issue that was discussed on IRC recently. > >> why not move ceph tests from ceph.git into ceph-qa-suite, Ideally this is >> not an issue with any major release, jewel workunits are supposed to work >> with jewel ceph, > > That would not help, because you still have the same issue of changes > in ceph being out of step with changes in ceph-qa-suite. > > I think your misconception here is to assume that running tests from > the same branch as the code is sufficient. You need to run tests from > the same *revision*. Otherwise, when I add a new test to reproduce a > bug, and fix the bug at the same time, then you will get failures if > you run the newer commit against the older code. > > John > >> only in rare cases it may cause some issue but easily fixable. All the dev >> runs can point to any qa-sha using the cli option so much less of an issue >> in rare case. > > > >>> >>> We could still enable folks running test changes without having to >>> rebuild ceph packages: the suite sha1 selected when running a >>> teuthology suite could still be different from that used for >>> installing ceph, it's just that it would fetch that sha1 from the ceph >>> repo instead of from a separate repo. >>> >>> John >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- Abhishek -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
