On Tue, 2 Jun 2015 07:02:06 -0400 (EDT) Kamil Paral <kparal@xxxxxxxxxx> wrote: > Hello, > > I have added some notes below. > > > The other thing to settle is what we actually want to report. Matt > > provided the following guidelines: > > > > - the current state of the subproject > > - future plans > > - things the team needs from the rest of the project > > - any blockers we can help unblock > > - big resource requests? > > > > Current state > > ------------- > > > > I'd say we have a fairly solid process in place now for release > > validation, and everyone's probably more or less familiar with it. > > We also have the update testing process nailed down, though I still > > think we can improve it in some ways, and we *still* want Bodhi > > 2.0 :) > > > > We have taskotron running for some package tests and openQA for some > > installation validation tests. openQA is I think still running on > > openSUSE boxes, but we have some work done on containerization to > > potentially move them to Fedora hosts (and I've also been working on > > packaging the openQA bits for Fedora). > > > > I think we might still be lacking a bit in terms of Change testing; > > there's a personpower issue there, though. We usually wind up only > > cherrypicking the most obviously potentially disruptive Changes for > > testing, and usually from the perspective of 'does this break the > > release'; we rarely manage to find the time to test Changes *in > > themselves* and help make sure they work well. > > I think that is a good summary. > > To provide more background for those not familiar with what has been > happening lately, we used openQA (an openSUSE project) in Fedora 22 > cycle to test about 20-25 different installation test cases, using > the tester name 'coconut'. We don't unfortunately have this system > running in public (even though Adam's development machine was), but > the pass results were sent into our wiki, and the failures were > examined manually. I think this was quite a success and it saved us a > lot of manual work. It also detected some issues, even though most of > those issues were detected manually as well. But the main benefit > here is, I believe, that we can skip a lot of repetitive manual > testing (because we know it's covered by openQA) and focus on > handling with real bugs, do more exploratory testing, etc. > > On the Taskotron front, led mainly by Tim and Martin in the recent > days, there has been some improvements and a lot of bug fixes. They > are mostly hidden under the hood, but one thing I would mention was > enabling tasks to store "task artifacts" - basically any files useful > for later review. This is currently not yet exposed in the ResultsDB > UI, but the rest of the code is there. Another invisible area which > has been improved is that we've been working our way towards > disposable test clients. Those will allow us to run destructive > checks, and potentially any third-party checks/tasks in the future. > This is by no means finished yet. This sounds right to me. Several bugfixes, some improvements but most of them aren't immediately obvious if you're not looking for them. At the moment, working on laying the groundwork for our future plans. > > Future plans > > ------------ > > > > We have a fairly well-defined roadmap for Taskotron development, I > > believe; we could broaden its actual test coverage, but I think we > > still have some infrastructure work that probably needs nailing down > > before we can focus on that, particularly in terms of the disposable > > test client work, and also I believe in terms of results management. > > Disposable test clients are now our current priority in Taskotron. We > also work on emitting fedmsg notifications, we need that for > integration with Bodhi2. Another middle-term feature could be, I > think, finishing up task artifacts to show up in ResultsDB and then > using that functionality to displaying only relevant parts of test > logs to maintainers (e.g. pidgin maintainer should only see test > output relevant for pidgin, and not also everything else) - the last > part is actually already implemented, but the dots are not connected > yet. I don't disagree with what kamil wrote here but I'd like to re-frame and reword it a little. Taskotron Future Plans ====================== The biggest long term goal for Taskotron right now is what I'm calling dist-git style tasks. The big idea here is to enable packagers to have tasks run for their packages by just pushing changes to git, similar to how dist-git works for packages (we may end up using the same repos but that's TBD). In order to enable this long term goal, we've broken it up into several smaller chunks: - Disposable Clients: One of the best ways to provide a good, repeatable platform for tasks is to use VMs spawned from a common image which are created before every task and destroyed after every task. This method is what we're calling "Disposable Clients" and is the current focus of our development activity - Finding a Home for the Tasks: Do we use dist-git? Do we use a separate git repository like dist-git? Find a home for the tasks and figure out the basic methods by which everything will work. - Locating some "guinea pig" volunteer maintainers Start small to make sure that everything is working like we plan - finding volunteers who are willing to help us iron out the kinks should make the eventual roll-out to all Fedora packages smoother The current plan is to have all this at least in stg by the end of this calendar year. I hope that things will be done faster than that but I choose to be conservative in my estimates to cover for when things go wrong. The best tools are the ones that people don't overtly notice and having the smoothest possible rollout helps there. Another somewhat related longer term goal for Taskotron is to enable Fedora contributors to write and submit tasks without much intervention from us. That will require most of the things for dist-git style tasks in addition to: - Rework our triggering mechanism That code was meant to live just long enough for us to figure out what we really needed and it's not really capable of fulfilling this role of allowing non-maintainers to schedule or add tasks. We knew this was going to be needed eventually. After that is beyond what my crystal ball can see clearly. There are plenty of things that we could focus on including (in no particular order) beaker integration, getting more strict build/update gating in place, supporting EPEL, supporting rawhide, supporting more testing types ... the list is long enough to keep us busy for years :) That being said, we'll cross that bridge once we get there - who knows what'll change before then and I don't see a point in spending energy to plan that far out given how quickly things change. I do expect the rate of change to increase once we get the currently planned things into place. Other Automation Plans ====================== We're still making slow progress on getting a beaker deployment in place. That should allow us to do more types of testing - especially bare metal testing, ks install testing and a few other things that don't fit well into Taskotron's current execution model. This is being done on a best-effort basis right now, so I have no specific ETA for a full production system but the staging setup should be done before too long. <snip> > > > > Things we need > > -------------- > > > > Kinda tied in to the above: Bodhi 2.0, and more efficient releng > > processes, are the big two I can think of - faster update pushes and > > composes. I think tflink and kparal are in a better position than I > > to know about any major resources we need. > > I don't say we "need" Bodhi2, but it would definitely help us to get > rid of some very nasty code (pushing comments to Bodhi), probably > improving speed and reliability of our checks considerably (we often > crash on Bodhi server errors). > > Another thing that would help a lot, I think, are some improvements > in the TC/RC engineering process, for example sending fedmsgs after a > TC is complete, or having a better structure metadata describing this > compose (for example, a json structure allowing us to see what > different ISO images are available for Server product, their > filepaths and types - DVD, netinst). We have a lot of black magic for > this already, and it mostly works, but such improvements would > simplify the code a lot of make it more reliable, especially when > some changes are introduced. > > For Taskotron itself, I don't have a feeling we're blocked on > something or urgently need something done, but from the > infrastructure point of view, Tim will definitely be able to provide > much better details. Yeah, I don't have anything specific and immediate for Taskotron or other automation. More experienced devs or devops/admin folks would probably help us get done a bit faster but that's not a requirement in any way, shape or form. Eventually, I'd like to find some volunteers amongst Fedora maintainers for the "guinea pig" role as we get dist-git style tasks rolled out but that's not an immediate need and I hope that finding a few folks who are willing to help us iron out the kinks won't be too hard. Tim
Attachment:
pgphcaaUCjFZy.pgp
Description: OpenPGP digital signature
-- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
