Congratulations guys, this is awesome work. I know it has been painful and I have a rough idea about the level of effort involved. Looking over the transition, this is truly solid work, and I am confident it will carry (and exceed) the packager's needs for at least another 13 releases or so... :-) Oh, yeah, and Jesse - thank you, for now people can finally stop mumbling blames my way. ;) Cristian Gafton On Thu, Jul 29, 2010 at 8:55 PM, Jesse Keating <jkeating@xxxxxxxxxx> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > This evening we opened up dist-git for business. Dist-git is our git > based replacement for dist-cvs, the source control system we were using > for our package specs, patches, and source files. This has been a long > time coming and a massive effort. I want to take a little time here to > outline what we've done and where we are going. > > First a brief outline of how our CVS system worked. CVS is a daemon of > sorts, and all repos typically live within a single CVSROOT. Within > this CVSROOT we had an 'avail' system to make use of, that we could > populate with data from Fedora Account System and dump into this file. > Avail worked on path names, relative to the CVSROOT. Since we used > directories for each Fedora release as pseudo branches we could set > avail info on each release "branch". CVS also used some filesystem > symlink tricks to create a "common" subdir for every package module, and > this is where we stuffed common scripts and Makefile content. Pretty > clever on one hand, we can make updates to the make system without > touching every single package, but it is pretty hackish and we had > constant issues where somebody would attempt an action using old common > content and stuff would fall over. > > Now we look at git. Git is for the most part a daemonless system. Each > repository is completely stand alone and generally does not require any > other infrastructure to be useful. You can interact with a repository > directly on the filesystem using /usr/bin/git or you can interact with > it through say ssh, again using /usr/bin/git (your local /usr/bin/git > will call a remote /usr/bin/git). Generally there is no running daemon > to connect to and authenticate with. Basic authentication of who can > check out what and commit where with git happens at the filesystem > permissions level. One twist with this is that with git, we wanted to > use real branches within a package repo to reflect the different > Fedora/EPEL releases. In repo branches are not reflected with path > names that we could set filesystem ACLs upon, so this posed a problem > for our conversion. > > Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an > addon system to git that provides ACL functionality including different > rights for different branches within a given repository, and more! By > using gitolite as a replacement for /usr/bin/git when a user connects to > our git server we can again utilize the information we have within the > Fedora Package Database and properly allow / restrict changes on > specific branches for specific developers. The gitolite upstream ( > Sitaram Chamarty, sitaram@xxxxxxxxxxx ) has been fantastically > responsive to our needs, which are admittedly a little unique. We have > a very large set of repositories (over 10.5K) and a largish number of > contributors (1050). The combination of the two leads to a very very > large and complex ACL structure that at first broke the system quite > badly. Upstream was very quick to create a "bigconfig" method of > compiling the ACLs without crashing the box. Our other unique needs > involve having individual accounts for each committer instead of a > shared account with a large list of allowed SSH keys. Add to that some > of our accounts need to be able to ssh shell into the git server for > administrative duties. Throughout our trials and testing with gitolite > every time we've ran into some issue that just didn't fit the mold, > Sitaram has been there with a smile and a fix. At this point our > production server is a whopping one line different from current gitolite > upstream. This is a fantastic win for us, for our sustainability, and > for the next large group that wants to make use of gitolite. > > Once we had a plan for ACLs and for branches, we had to decide if we > were going to replace the Make system and with what. I had never been a > fan of Make, it was entirely too difficult to modify and innovate with. > Since I was the one pushing this transition forward, I decided to write > a new tool in my favorite language, python. The fedpkg tool was born > and took off. fedpkg was born around January 4th, 2010 and has since > grown into 1,523 (via sloccount) lines of code. While far from > complete, it is a great start (if I do say so myself!) at replacing the > make system. Because it is written in python (or maybe just !Make) it > seems to be easier to contribute to, and I've already gotten a number of > contributions. More will come as it starts to be more widely used. The > biggest challenge with fedpkg is removing the need to update something > on the end user's system every single time we added a new Fedora release > and changed what happens when you build for rawhide. I'll spare you the > details but I'm fairly happy with what we have. The end result should > be far fewer misfires and end user confusion. > > The last major piece of the puzzle was how to actually convert the > existing CVS repositories, including the fun pseudo branches, into git > repositories. I tried a number of options over the years (I've been > working on this off and on for nearly 4 years!) ranging from the built > in git cvsimport to git-svn to parsecvs and a few others. In the end, > we took a page from the gnome project and used parsecvs ( > http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of > our repositories. There were a few that gave parsecvs fits and recent > versions of git cvsimport were able to handle them. The git system is > fantastic enough that we were able to merge our pseudo cvs branches into > actual git branches complete with a real shared history, but again I'll > spare you the details of the scripting to do this. All but the kernel > repo seems to have converted successfully which is a pretty good > success rate in my book. We may yet get the kernel converted, but in > the interest of time we opted to start fresh with dist-git for now. > > Without the help of many others, this project would never have gotten > done. Folks helped out with Koji modifications, with fedpkg > contributions, with repeated testing of attempted conversions, with > logic checking of my plans, of helping me understand more of git > internals and deciphering error output, and most of all with being > patient while we worked through the transition and very positive along > the way. Things will be bumpy over the next few weeks as we really > start putting distgit to the test. No amount of staging and testing can > really replace production use. There will be many more updates to > fedpkg as bugs are found and fixed and features are contributed. Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ). > > Once again I want to thank everybody who helped out and for all the > (continued) patience! I'll be available via email and IRC as much as > possible the next few days to help anybody with dist-git issues. Look > for Oxf13 on freenode. Happy gitting! > > - -- > Jesse Keating > Fedora -- Freedom² is a feature! > identi.ca: http://identi.ca/jkeating > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.14 (GNU/Linux) > Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy > VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs > =fBGf > -----END PGP SIGNATURE----- > _______________________________________________ > devel-announce mailing list > devel-announce@xxxxxxxxxxxxxxxxxxxxxxx > https://admin.fedoraproject.org/mailman/listinfo/devel-announce > -- Cristian Gafton <gafton@xxxxxxxxx> -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
