----- Original Message ----- > From: "Ralph Bean" <rbean@xxxxxxxxxx> > To: "Fedora Infrastructure" <infrastructure@xxxxxxxxxxxxxxxxxxxxxxx> > Sent: Tuesday, May 28, 2013 6:45:43 AM > Subject: Re: areas where we can invest in automation? > > On Tue, May 28, 2013 at 08:33:22AM -0400, Matthew Miller wrote: > > I was asked (with my Red Hat hat on) to put together a little report on > > areas in Fedora which could be improved with an investment in better > > automation. From what I'm working on myself, I'm aware of the gigantic need > > in the cloud images production process, and I've been keeping an eye on Tim > > Flink's autoqa revamp ideas. I expect there are others, because I know from > > my previous jobs that there's always a balance between building > > condiment-passing machines and just _passing the salt_. [1] Are there other > > things which could be made better if only someone came up with the spare > > time and resources to do the work? > > Hi! > > Some observations/ideas: > > 1) The packager workflow is pretty tedious. There has been some > improvement to it, but more can be done. Things like > fedora-review and fedora-create-review (and bodhi!) are a huge > help. But there are plenty of inefficient "blocking" points in the > process. > > For instance, once a new package is approved, only then does the > submitter declare what branches they want with an scm admin > request. They then wait for an scm admin to declare that they > have created their branches, and then wait for a cronjob to run > that gives them permission to push on those branches (manually). > They then wait for their koji builds to finish to (manually) submit > bodhi updates. > > It would be nice if we could automate that whole process -- once a > package is approved, if there were a "make-it-so" button that > required no further intervention from the packager (but still > required the keen eye of an scm admin). > > There are further sequences down the pipeline like requesting that > packages in testing be pushed to stable, but there are good > arguments against automating that. > > 2) Continuous deployment for infrastructure. It has been tossed > around in IRC, possibly at FUDCon as well. If application > developers could "git push" on the develop branch and have those > changes automatically roll out to our staging infrastructure -- > that would save a lot of time. Packaging our apps, building rpms, > signing them, copying them to our infra yum repos, rebuilding those > repos, clearing the cache on the target machines, performing a yum > update <-- that process is cumbersome. > > I suspect that the "release only when we have accumulated enough > changes to warrant enduring the burdensome release process" mode > of deployment (as opposed to "release early, release often") also > poses somewhat of a barrier to new contributors. They contribute > a patch.. nice! When does it go live? When one of our > overstretched sysadmin-mains can get around to it (it is required > that one of them sign the package). This would be super. And yes - I think that the "instant gratification" combined with having to know what is likely a lot less process seems like it would yield additional contributions. Do we have tiers for apps according to their "must stay-up" level - things that are critical vs. NTH? It seems like something where we could probably pick a non-critical app or two and work out what the process would be and basically take that process for a in-production-test-run. > > Caveat #2.1: > There are some ways around this. Individuals can get around the > requirement of having a sysadmin-main touch their test release by > installing their rpm directly on the target machine. They still > have to jump through some hoops to make it happen. > > Caveat #2.2: > This is one of the reasons we put so much work into our private > cloud (dev nodes). There is no barrier there for teams to set up > their own continuous deployment mechanism. This meets most needs, > but we don't have a way to iterate rapidly on some of the more > important pieces of our infrastructure. Apps/services that > interact with each other don't quite work out on isolated cloud > nodes. The bodhi masher? Koji? fedmsg? mirror manager? We > can't necessarily test those on dev nodes (and some we can't test > in staging -- resolving this down the road would save some > headaches). So I'm definitely treading into "woefully technically unedumacated" territory - but could we not just take snapshots of the associated apps/services/databases, test whatever changes are incoming against the snapshots, and if it passes, push the changes to production and dump the tested nodes? (I realize this is simplifying things - we might have changes that require changes in both places, etc - but just wondering if I'm totally off-base here.) Or is this just a matter of "the infra things aren't running in the same place"? -robyn > > _______________________________________________ > infrastructure mailing list > infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > https://admin.fedoraproject.org/mailman/listinfo/infrastructure _______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
