On 07/30/2010 03:02 PM, Jon Ciesla wrote: > On 7/29/2010 10:55 PM, Jesse Keating wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> This evening we opened up dist-git for business. Dist-git is our git >> based replacement for dist-cvs, the source control system we were using >> for our package specs, patches, and source files. This has been a long >> time coming and a massive effort. I want to take a little time here to >> outline what we've done and where we are going. >> >> First a brief outline of how our CVS system worked. CVS is a daemon of >> sorts, and all repos typically live within a single CVSROOT. Within >> this CVSROOT we had an 'avail' system to make use of, that we could >> populate with data from Fedora Account System and dump into this file. >> Avail worked on path names, relative to the CVSROOT. Since we used >> directories for each Fedora release as pseudo branches we could set >> avail info on each release "branch". CVS also used some filesystem >> symlink tricks to create a "common" subdir for every package module, and >> this is where we stuffed common scripts and Makefile content. Pretty >> clever on one hand, we can make updates to the make system without >> touching every single package, but it is pretty hackish and we had >> constant issues where somebody would attempt an action using old common >> content and stuff would fall over. >> >> Now we look at git. Git is for the most part a daemonless system. Each >> repository is completely stand alone and generally does not require any >> other infrastructure to be useful. You can interact with a repository >> directly on the filesystem using /usr/bin/git or you can interact with >> it through say ssh, again using /usr/bin/git (your local /usr/bin/git >> will call a remote /usr/bin/git). Generally there is no running daemon >> to connect to and authenticate with. Basic authentication of who can >> check out what and commit where with git happens at the filesystem >> permissions level. One twist with this is that with git, we wanted to >> use real branches within a package repo to reflect the different >> Fedora/EPEL releases. In repo branches are not reflected with path >> names that we could set filesystem ACLs upon, so this posed a problem >> for our conversion. >> >> Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an >> addon system to git that provides ACL functionality including different >> rights for different branches within a given repository, and more! By >> using gitolite as a replacement for /usr/bin/git when a user connects to >> our git server we can again utilize the information we have within the >> Fedora Package Database and properly allow / restrict changes on >> specific branches for specific developers. The gitolite upstream ( >> Sitaram Chamarty, sitaram@xxxxxxxxxxx ) has been fantastically >> responsive to our needs, which are admittedly a little unique. We have >> a very large set of repositories (over 10.5K) and a largish number of >> contributors (1050). The combination of the two leads to a very very >> large and complex ACL structure that at first broke the system quite >> badly. Upstream was very quick to create a "bigconfig" method of >> compiling the ACLs without crashing the box. Our other unique needs >> involve having individual accounts for each committer instead of a >> shared account with a large list of allowed SSH keys. Add to that some >> of our accounts need to be able to ssh shell into the git server for >> administrative duties. Throughout our trials and testing with gitolite >> every time we've ran into some issue that just didn't fit the mold, >> Sitaram has been there with a smile and a fix. At this point our >> production server is a whopping one line different from current gitolite >> upstream. This is a fantastic win for us, for our sustainability, and >> for the next large group that wants to make use of gitolite. >> >> Once we had a plan for ACLs and for branches, we had to decide if we >> were going to replace the Make system and with what. I had never been a >> fan of Make, it was entirely too difficult to modify and innovate with. >> Since I was the one pushing this transition forward, I decided to write >> a new tool in my favorite language, python. The fedpkg tool was born >> and took off. fedpkg was born around January 4th, 2010 and has since >> grown into 1,523 (via sloccount) lines of code. While far from >> complete, it is a great start (if I do say so myself!) at replacing the >> make system. Because it is written in python (or maybe just !Make) it >> seems to be easier to contribute to, and I've already gotten a number of >> contributions. More will come as it starts to be more widely used. The >> biggest challenge with fedpkg is removing the need to update something >> on the end user's system every single time we added a new Fedora release >> and changed what happens when you build for rawhide. I'll spare you the >> details but I'm fairly happy with what we have. The end result should >> be far fewer misfires and end user confusion. >> >> The last major piece of the puzzle was how to actually convert the >> existing CVS repositories, including the fun pseudo branches, into git >> repositories. I tried a number of options over the years (I've been >> working on this off and on for nearly 4 years!) ranging from the built >> in git cvsimport to git-svn to parsecvs and a few others. In the end, >> we took a page from the gnome project and used parsecvs ( >> http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of >> our repositories. There were a few that gave parsecvs fits and recent >> versions of git cvsimport were able to handle them. The git system is >> fantastic enough that we were able to merge our pseudo cvs branches into >> actual git branches complete with a real shared history, but again I'll >> spare you the details of the scripting to do this. All but the kernel >> repo seems to have converted successfully which is a pretty good >> success rate in my book. We may yet get the kernel converted, but in >> the interest of time we opted to start fresh with dist-git for now. >> >> Without the help of many others, this project would never have gotten >> done. Folks helped out with Koji modifications, with fedpkg >> contributions, with repeated testing of attempted conversions, with >> logic checking of my plans, of helping me understand more of git >> internals and deciphering error output, and most of all with being >> patient while we worked through the transition and very positive along >> the way. Things will be bumpy over the next few weeks as we really >> start putting distgit to the test. No amount of staging and testing can >> really replace production use. There will be many more updates to >> fedpkg as bugs are found and fixed and features are contributed. Wiki >> pages will get filled out as knowledge of how to interact with dist-git >> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a >> good start ). >> >> > When I do fedpkg clone -B libfoo, to get the old-style layout, all the > branches are identical to devel. The branches on the standard clone > seem fine. I've not made any changes with the -B layout, but I thought > this should be corrected or deprecated. > > fedora-packager-0.5.1.0-1.fc13.noarch > > -J Oh, and since I neglected to say so, thanks a huge heaping ton to Jesse and everyone else involved in making this happen, I've been excited to see it happen, and I have to say after using it today that the improvements in speed and usability are better than I expected. Kudos all around. -J >> Once again I want to thank everybody who helped out and for all the >> (continued) patience! I'll be available via email and IRC as much as >> possible the next few days to help anybody with dist-git issues. Look >> for Oxf13 on freenode. Happy gitting! >> >> - -- >> Jesse Keating >> Fedora -- Freedom² is a feature! >> identi.ca: http://identi.ca/jkeating >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2.0.14 (GNU/Linux) >> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ >> >> iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy >> VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs >> =fBGf >> -----END PGP SIGNATURE----- >> _______________________________________________ >> devel-announce mailing list >> devel-announce@xxxxxxxxxxxxxxxxxxxxxxx >> https://admin.fedoraproject.org/mailman/listinfo/devel-announce >> > -- - in your fear, speak only peace in your fear, seek only love -d. bowie -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
