On Thu, Apr 23, 2020 at 01:36:48PM -0700, Kevin Fenzi wrote: > On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote: > > Good Morning Everyone, > > > > Kevin and I had a discussion on IRC the other day about mirroring the ansible > > repo. > > The options we considered are: > ...snip... > > > > For a first step I went with a third approach: a small python service that > > runs every 3 minutes (configurable): git fetch && git fsck (to ensure the git > > is in a correct state). It actually doesn't run every 3 minutes, it calls git, > > then wait for 3 minutes then calls git again and so on. So if git was ever to > > takes 4 minutes to run, there is no risk of the program stepping on its own > > toes. > > ok. > > I'm wondering if that might be overkill. > > Really if pagure.io blows up, we want to be able to look at the repo we > have on batcave01 and be able to still operate while we recover > pagure.io. If someone pushed or merged some commits in between the last > time we synced and blowup, someone should have them in their local repo > right? so we could just push them to the repo on batcave and be back > completly in sync, no? Someone will have the commits sure, but since it's a PR, you could be merging a couple of commits from me while I am asleep. If pagure goes down right after the merge, you will not have the commits on your fork and batcave may not have synced them, leading to these commits not being accessible. This isn't a situation we couldn't recover from, but one of the two repos will need to get a force-push once they are both available again. > but I guess it doesn't hurt to do every 3minutes, just some BW. I think the fedora-messaging client would reduce the BW usage, and I think I have it ready, so we may be able to go with it from the start. > Another option I mentioned the other day when we were talking, but I > didnt really explain well was that we could wrap ansible-playbook into a > wrapper that runs a git pull then ansible-playbook... so it would always > be in sync at first you run it. That could be messy tho if there's > another process doing pulls, etc. So I guess scrap that idea. > > > I would like to propose that install and configure this for a test. > > I propose that we keep /git/ansible.git as is and just have a > > /git/mirrors/ansible.git which will be the mirror from pagure. > > We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so > > /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror. > > I think that could be confusing. If we switch we should switch and make > all the old checkouts no longer work for pushing, or we are going to > cause a lot of confusion. I'd say move the old one to some .back name > and chmod 000 it. Works for me, I was thinking to keep it as some sort of backup that would be updated hourly or so and thus would give us a little breathing room in case of disaster (quickly diagnosed). I'm ok with your approach that the breathing room gained may be overkilled there (the chances that we diagnose an issue and think about stopping the service while we're trying to debug the issue and that the service hasn't synced in the mean time seems... low). > Also, there is the question about the existing git repo for > fedora-infrastructure on pagure. It has stuff in it now. Do we > completely replace it with the current ansible repo? Or do we stream all > the ansible commits on top of it so current checkouts would just be able > to pull? Or do we put it on another project. I'd really prefer to have > it on the same project our tickets are to avoid confusion. I kinda think > it might be nice to stream it on top of the existing repo so it's not > broken, but not sure how long or messy that would be. I was going for: https://pagure.io/Fedora-Infra/ansible/ which is marked as being the future location of our ansible repo and already has a copy of it. I had not realized we have content in https://pagure.io/fedora-infrastructure/ 's git. If we really want to have the ansible repo in that repo, the easiest may be to rebase these commits on the top of ansible and force push to that repo (so current ansible clone would pull fine once they've updated their urls=). > > I guess the lag time to get /git/mirrors updated will quickly be annoying (up to > > 3 minutes between when the PR is merged and when the playbook is updated), so > > if we like the workflow we will likely want to switch pretty quickly to a > > message-based service and then we could keep the current cron-like service to > > run hourly and update /git/ansible.git which would then become some kind of > > backup. > > So, we already have fedmsg-hub listening on batcave01 for fas messages > (which no longer happen). We could just reuse that? Or a > fedora-messaging version of it I guess would be better. Almost there :) > > What do you think? > > FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't > > do we want to try this sooner?) > > Wait until after freeze. We have waited this long and we still have > release things to do in ansible. I don't want us to mess up release for > this. But we can get all our birds in a line to do it right after > freeze. > > > If we like the idea, I'll start looking into the fedora-messaging based > > consumer, using the approach from the current service, it should be pretty > > straight forward (I'll re-use the same project for it). > > So this would just listen for commit / pr merge and do a git pull / > sync? Listens to commits in pagure.io and pull when it sees some in the ansible repo. Pierre
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
