Re: The move to git!

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Congratulations guys, this is awesome work. I know it has been painful
and I have a rough idea about the level of effort involved. Looking
over the transition, this is truly solid work, and I am confident it
will carry (and exceed) the packager's needs for at least another 13
releases or so... :-)

Oh, yeah, and Jesse - thank you, for now people can finally stop
mumbling blames my way. ;)

Cristian Gafton

On Thu, Jul 29, 2010 at 8:55 PM, Jesse Keating <jkeating@xxxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This evening we opened up dist-git for business.  Dist-git is our git
> based replacement for dist-cvs, the source control system we were using
> for our package specs, patches, and source files.  This has been a long
> time coming and a massive effort.  I want to take a little time here to
> outline what we've done and where we are going.
>
> First a brief outline of how our CVS system worked.  CVS is a daemon of
> sorts, and all repos typically live within a single CVSROOT.  Within
> this CVSROOT we had an 'avail' system to make use of, that we could
> populate with data from Fedora Account System and dump into this file.
> Avail worked on path names, relative to the CVSROOT.  Since we used
> directories for each Fedora release as pseudo branches we could set
> avail info on each release "branch".  CVS also used some filesystem
> symlink tricks to create a "common" subdir for every package module, and
> this is where we stuffed common scripts and Makefile content.  Pretty
> clever on one hand, we can make updates to the make system without
> touching every single package, but it is pretty hackish and we had
> constant issues where somebody would attempt an action using old common
> content and stuff would fall over.
>
> Now we look at git.  Git is for the most part a daemonless system.  Each
> repository is completely stand alone and generally does not require any
> other infrastructure to be useful.  You can interact with a repository
> directly on the filesystem using /usr/bin/git or you can interact with
> it through say ssh, again using /usr/bin/git (your local /usr/bin/git
> will call a remote /usr/bin/git).  Generally there is no running daemon
> to connect to and authenticate with.  Basic authentication of who can
> check out what and commit where with git happens at the filesystem
> permissions level.  One twist with this is that with git, we wanted to
> use real branches within a package repo to reflect the different
> Fedora/EPEL releases.  In repo branches are not reflected with path
> names that we could set filesystem ACLs upon, so this posed a problem
> for our conversion.
>
> Enter gitolite.  Gitolite ( http://github.com/sitaramc/gitolite )is an
> addon system to git that provides ACL functionality including different
> rights for different branches within a given repository, and more!  By
> using gitolite as a replacement for /usr/bin/git when a user connects to
> our git server we can again utilize the information we have within the
> Fedora Package Database and properly allow / restrict changes on
> specific branches for specific developers.  The gitolite upstream (
> Sitaram Chamarty, sitaram@xxxxxxxxxxx ) has been fantastically
> responsive to our needs, which are admittedly a little unique.  We have
> a very large set of repositories (over 10.5K) and a largish number of
> contributors (1050).  The combination of the two leads to a very very
> large and complex ACL structure that at first broke the system quite
> badly.  Upstream was very quick to create a "bigconfig" method of
> compiling the ACLs without crashing the box.  Our other unique needs
> involve having individual accounts for each committer instead of a
> shared account with a large list of allowed SSH keys.  Add to that some
> of our accounts need to be able to ssh shell into the git server for
> administrative duties.  Throughout our trials and testing with gitolite
> every time we've ran into some issue that just didn't fit the mold,
> Sitaram has been there with a smile and a fix.  At this point our
> production server is a whopping one line different from current gitolite
> upstream.  This is a fantastic win for us, for our sustainability, and
> for the next large group that wants to make use of gitolite.
>
> Once we had a plan for ACLs and for branches, we had to decide if we
> were going to replace the Make system and with what.  I had never been a
> fan of Make, it was entirely too difficult to modify and innovate with.
>  Since I was the one pushing this transition forward, I decided to write
> a new tool in my favorite language, python.  The fedpkg tool was born
> and took off.  fedpkg was born around January 4th, 2010 and has since
> grown into 1,523 (via sloccount) lines of code.  While far from
> complete, it is a great start (if I do say so myself!) at replacing the
> make system.  Because it is written in python (or maybe just !Make) it
> seems to be easier to contribute to, and I've already gotten a number of
> contributions.  More will come as it starts to be more widely used.  The
> biggest challenge with fedpkg is removing the need to update something
> on the end user's system every single time we added a new Fedora release
> and changed what happens when you build for rawhide.  I'll spare you the
> details but I'm fairly happy with what we have.  The end result should
> be far fewer misfires and end user confusion.
>
> The last major piece of the puzzle was how to actually convert the
> existing CVS repositories, including the fun pseudo branches, into git
> repositories.  I tried a number of options over the years (I've been
> working on this off and on for nearly 4 years!) ranging from the built
> in git cvsimport to git-svn to parsecvs and a few others.  In the end,
> we took a page from the gnome project and used parsecvs (
> http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of
> our repositories.  There were a few that gave parsecvs fits and recent
> versions of git cvsimport were able to handle them.  The git system is
> fantastic enough that we were able to merge our pseudo cvs branches into
> actual git branches complete with a real shared history, but again I'll
> spare you the details of the scripting to do this.  All but the kernel
> repo  seems to have converted successfully which is a pretty good
> success rate in my book.  We may yet get the kernel converted, but in
> the interest of time we opted to start fresh with dist-git for now.
>
> Without the help of many others, this project would never have gotten
> done.  Folks helped out with Koji modifications, with fedpkg
> contributions, with repeated testing of attempted conversions, with
> logic checking of my plans, of helping me understand more of git
> internals and deciphering error output, and most of all with being
> patient while we worked through the transition and very positive along
> the way.  Things will be bumpy over the next few weeks as we really
> start putting distgit to the test.  No amount of staging and testing can
> really replace production use.  There will be many more updates to
> fedpkg as bugs are found and fixed and features are contributed.  Wiki
> pages will get filled out as knowledge of how to interact with dist-git
> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
> good start ).
>
> Once again I want to thank everybody who helped out and for all the
> (continued) patience!  I'll be available via email and IRC as much as
> possible the next few days to help anybody with dist-git issues.  Look
> for Oxf13 on freenode.  Happy gitting!
>
> - --
> Jesse Keating
> Fedora -- Freedom² is a feature!
> identi.ca: http://identi.ca/jkeating
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy
> VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs
> =fBGf
> -----END PGP SIGNATURE-----
> _______________________________________________
> devel-announce mailing list
> devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
> https://admin.fedoraproject.org/mailman/listinfo/devel-announce
>



-- 
Cristian Gafton <gafton@xxxxxxxxx>
-- 
devel mailing list
devel@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/devel



[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]
  Powered by Linux