Hi, On Wed, 22 Aug 2018 at 16:02, Emil Velikov <emil.l.velikov at gmail.com> wrote: > On 22 August 2018 at 12:44, Daniel Vetter <daniel.vetter at ffwll.ch> wrote: > > I think it's time to brainstorm a bit about the gitlab migration. Basic reasons: > > > > - fd.o admins want to deprecate shell accounts and hand-rolled > > infrastructure, because it's a pain to keep secure&updated. > > > > - gitlab will allow us to add committers on our own, greatly > > simplifying that process (and offloading that task from fd.o admins). > > Random thought - I really wish the admins spoke early and louder about issues. > From infra to manpower and adhoc tool maintenance. I thought I mostly had it covered, but fair enough. What knowledge are you missing and how should it best be delivered? One first-order issue is that as documented at https://www.freedesktop.org/wiki/AccountRequests/ creating accounts requires manual admin intervention. You can also search the 'freedesktop.org' product on Bugzilla to see the amount of time we spend shuffling around GPG & SSH keys, which is about the worst possible use of my time. Many other people have had access to drive the ancient shell-script frontend to LDAP before, but for some reason they mostly aren't very enthusiastic about doing it all the time. In the mesa-dev@ thread about Mesa's migration, which is linked from my blog post, I went into quite a lot of detail about why Jenkins was not suitable to roll out across fd.o globally. That knowledge was gained from trial & error, which was a lot of time burnt. The end result is that we don't have any CI, except if people hang Travis/AppVeyor off GitHub mirrors. You've personally seen what's involved in setting up Git repository hooks so we can build docs. We can't give access to let people work on those, without giving them direct access to the literal Git repository itself on disk. The hooks mostly involve bespoke sets of rsync jobs and hand-managed SSH credentials and external services to build docs and so on and so forth. None of this is accessible to a newcomer who wants to make a non-code contribution: you have to find someone with access to the repo to go bash some shell scripts directly and hope you didn't screw it up. None of this is auditable, so if the repo mysteriously gets wiped, then hopefully there are some backups somewhere. But there definitely aren't any logs. Luckily we prevent most people from having access to most repositories via a mandatory 'shell' which only allows people to push Git; this was written by hand by us 12 years ago. What else? Our fork of Patchwork was until recently based on an ancient fork of Django, in a bespoke unreproducible production environment. Bugzilla is patched for spam and templates (making upgrades complex), yet we still have a surprising amount of spam pass through. There's no way to delete spam, but you have to manually move every bug to the 'spam' group, then go through and delete the user which involves copying & pasting the email and a few clicks per user. Mailman is patched to support Recaptcha, bringing more upgrade fun. ikiwiki breaks all the time because it's designed to have the public-access web server on the same host as the writeable Git repositories. Our servers are several years old and approaching life expiry, and we have no more spare disk. You can see in #freedesktop IRC the constant network connectivity issues people have to PSU almost every day. Our attempts to root-cause and solve those have got nowhere. I could go on, but the 'moved elsewhere' list in https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2 indicates that we do have problems to solve, and that changing peoples' SSH keys for them isn't the best way for us to be spending the extremely limited time that we do have. > > For the full in-depth writeup of everything, see > > > > https://www.fooishbar.org/blog/gitlab-fdo-introduction/ If you haven't seen this, or the post it was linked from, they would be a good start: https://lists.freedesktop.org/archives/freedesktop/2018-July/000370.html There's also the long thread on mesa-dev a long time back which covers a lot of this ground already. > > - Figuring out the actual migration - we've been adding a pile of > > committers since fd.o LDAP was converted to gitlab once back in > > spring. We need to at least figure out how to move the new > > accounts/committers. > > As a observer, allow me to put some ideas. You've mostly covered them > all, my emphasis is to seriously stick with _one_ thing at a time. > Attempting to do multiple things in parallel will end up with > sub-optimal results. > > - (at random point) cleanup the committers list - people who have not > contributed in the last 1 year? libdrm was previously covered under the Mesa ACL. Here are the other committer lists, which you can see via 'getent group' on annarchy or another machine: amdkfd:x:922:fxkuehl,agd5f,deathsimple,danvet,jazhou,jbridgman,hwentland,tstdenis,gitlab-mirror,rui drm-meson:x:936:narmstrong drm:x:940:airlied,danvet drm-intel:x:920:daniels,ickle,danvet,jwrdegoede,pzanoni,mlankhorst,ideak,vivijim,vsyrjala,jani,miku,mattrope,tursulin,dolphin,llandwerlin,lyudess,dpandiya,mthierry,mdnavare drm-misc:x:932:daniels,daenzer,anholt,agd5f,ickle,deathsimple,kraxel,danvet,jwrdegoede,mlankhorst,tagr,mperes,vivijim,dvdhrm,vsyrjala,pinchartl,jani,evelikov,lynxeye,dolphin,sumits,architt,seanpaul,llandwerlin,lyudess,padovan,narmstrong,hwentland,bbrezillion,shawnguo,ahajda,lukas,vinceab,benjamin.gaignard,patrik,pcornu,notro,linusw,mripard,hjc,anushas,mmind,hyunk,tomba,jsarha,wens,andr2000,alexg1,dliviu,ayankh,dlech,mdnavare,shashank,ph5 > - setup drm group, copy/migrate accounts - one could even reuse the > existing credentials Yes, dealing with accounts is covered here: https://gitlab.freedesktop.org/freedesktop/freedesktop/wikis/home#how-do-i-createaccess-my-user-account That page is linked from the blog post as well as the freedesktop@ post. > - move git repos to gitlab, the push URL change, cgit mirror > preserves the normal fetch ones as well as PW hooks Yes, this is what happens. The old repos give you a long message explaining how to set up your GitLab account and change your Git remote if you try to push to them. > - work out how new accounts are handled - still in bugzilla, otherwise > > At this stage only workflow change is a) once-off account setup and b) > pushURL update > As a follow-up one can setup anything fancy. > - migrate PW/other hooks > - migrate bugs - if applicable > - add new hooks - DRM docs, other > - etc > > > > - Similar, maintainer-tools needs to move. We probably want to move > > all the dim maintained kernel repos in one go, to avoid headaches with > > double-accounts needed for committers. > > One should be able to create a separate repo for these. And then either: > - one by one add the required features into the gitlab MR machinery, > - or, wire the execution in pre/post merge stage. > > IIRC there are some upstream requests about the former. He's just talking about migrating repository hosting (where people push), not opening up for MRs (how people review code); see the reply to Jani upthread for more detail. > > - CI, linux-next and everyone else should be fine, since the > > cgit/non-ssh paths will keep working (they'll be read-only mirrors). > > Need to double-check that with everyone. > > > > - Some organization structure would be good. > > > > https://cgit.freedesktop.org/drm > > > > libdrm won't be part of the gitlab drm group because that's already > > moved under mesa (and you can't symlink/mulit-home anymore on gitlab): > > > > https://gitlab.freedesktop.org/mesa/drm > > > > But there's also drm_hwcomposer, which we might want to migrate into > > drm too - gitlab requires a containing group, and > > drm_hwcomposer/drm_hwcomposer is a bit silly. > > > It did strike me a lot when drm_hwcomposer/drm_hwcomposer was > introduced. Fortunately moving repos in gitlab is reasonably pain-free It is, but if they don't share an ACL and the only benefit is cosmetic, then ... meh? Weston doesn't live in drm/ either and that's OK. Cheers, Daniel
