On Thu, Nov 24, 2022 at 05:28:22PM -0000, Francois Andrieu wrote: > Hi everyone! Hello. > The Websites & Apps team is currently working on rewriting all major Fedora websites (such as getfedora, spins & alt) and I believe this is a good opportunity to revisit the current deployment workflow and try to make it simpler. Indeed. I have often thought about this workflow. ;) > Currently, websites are being built in Openshift, with a cronjob running every hour that fetches code from git, builds it, then saves it to an NFS share. > That same NFS share is also mounted on sundries01, which exposes it through rsync for proxies to sync it, then serve it to the world. > > I have a few solutions in mind to replace that, and I would like your input on them. > > A) Full Openshift > We can build, and deploy the websites directly in Openshift, and serve it from here. Just like silverblue.fp-o. > While this is probably the most straightforward solution, it has one major downside: if Openshift is unavailable for any reason, our major websites become also unavailable. > I believe this is why we are still using our proxies to host them, as such a scenario is unlikely to happen on every single one of them at the same time. Yep. That and also proxies are usually 'closer' to the end user and serving static content, so it's much faster network wise. ie, say a user in germany would just hit a german proxy and get the content fast, while if we moved it to openshift they would have to transit all the way over to the us and back to get that content. > B) Same as before, with a twist > We build on Openshift, but instead of going through NFS and sundries with rsync, we store the websites on S3 storage provided by Openshift, then we sync the proxies using `s3cmd sync`. I think this is definitely a improvement over the current setup. > C) Same as B, but with an external builder > We already build the new websites on Gitlab CI, and since the S3 gateway is accessible from the outside, we could just push the build artifacts to s3 directly from GitLab CI. Then sync the proxies from it. Or just pull from gitlab directly? Or does it expose that data in a way we could sync from? > D) Keep using the Openshift->Sundries->proxies workflow I'd like to move to something better once we decide whats worth trying. :) > E) Your solution here. Some more I have thought on: E) a twist on A. We build and serve in openshift, but we stick cloudfront in front of it. This would solve the speed problems, but still would have the openshift down issue. F) (this is a fun one :) How about looking into FCOS or RHEL for edge? In this model we would install ostree based vm's in the places we have proxies now and we would build the web content as a ostree ref and pull it from our registry (or quay.io). I think this would be fun, but probibly overkill/too much effort for just static content, but I thought I would throw it out there. > We could also improve B and C by adding fedora-messaging to the mix to trigger a proxy resync as soon as a new build is available instead of doing so every hour. +1 (although we should make sure not to thundering herd the endpoint if all proxies decided to sync at the same instant). > What do you all think? I like B... but possibly could be talked into C. The thing I don't like about C is that it has less visibility, if there was a problem, it would require someone from websites to fix, rather than possibly being something that anyone with access to openshift could fix. kevin
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
