I'm +1 to have this documented in developer guidelines
https://docs.fedoraproject.org/en-US/infra/developer_guide/
More replies inline.
On 18. 06. 22 4:58, Jon Stanley wrote:
Welcome to the hell at $DAYJOB :). Some comments below.....
On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi <kevin@xxxxxxxxx> wrote:
So, when moving openshift apps this week, I really came to the
conclusion we should look at writing up some guidelines for openshift
apps. Perhaps it could have 'MUST' items and 'SHOULD' items and
'SUGGESTION' items.
Yep, sounds good. We have something that we call the "big rules" which
fit some of the things here.
I know I am coming at this from the operations side of things, so please
do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is
running depends on when it was last built/deployed. (unless: see
below)
Yep. It's a bad practice to use "latest", especially for an image that
you didn't build. This leaves you with zero control over potentially
breaking changes
When I last checked, I noticed that we have plenty of apps where you are
not even able to find what they are using, because it's not specified in
the buildconfig. They are instead using some Dockerfile in the upstream
project, which
isn't bad, but it could cause confusion and it's hard to track.
- A number of apps use images elsewhere (quay.io, etc). We have very
little way to tell whats in those or how they are made or if anyone is
still making them.
This is an advantage that we have being a large enterprise - our
clusters have no Internet access and therefore all images come from a
registry that we control. Perhaps something similar would be valuable
here.
We have a few apps that rebuild/deploy on git changes, which is fine,
but doesn't do anything to keep their container env up to date/secure.
One thing that's possible to fix this is to use vulnerability scanning
products. We use a proprietary one, but just recently Red Hat
open-sourced Stackrox (a company that was acquired a bit ago -
https://stackrox.io). I haven't used either the upstream or downstream
(Advanced Cluster Security for Kubernetes) version of it (though I did
build the upstream version just to see that it worked, it's somewhat
difficult to build and AFAIK there's no distribution
mechanism/prebuilt images for it - they threw some source code over
the wall AFAICT and are looking to improve the build process "soon" so
that they can ship prebuilt images - i.e., the build currently
requires Docker and doesn't seem to work with podman-docker (I
tried.....)
We have a number of apps that tigger on imagestream changes, but as far
as I can tell thats only changes in their image ('bodhi-base') not the
base image that it was built from ('fedora:36'). I'd like to see us
track base images and build/deploy when one changes. If we do that then
:latest might be ok to use, since we know automation took care of
updating it for us.
Once you figure that out, let me know :). Bonus points if it doesn't
use imagestreams. You still don't want :latest though :).
There's a number of applications using ubi (rhel's base image), which I
think might be worth recommending, especially if we start build/deploy
on base image changes. Likely Fedora base images will change more
quickly.
Yep, this is one of our "big rules" that we prefer ubi-based images.
For internally built code, we require it and for vendor apps we prefer
it. Since I imagine that we don't have many "vendor" apps (except
maybe stuff like Redis, which already exists as a Red Hat packaged
downstream image which should likely be preferred), the amount of
non-ubi images should be minimal I would think.
Currently we have some apps that were setup and probibly have not
changes in years, happily running the same old container image since
there was no reason to redeploy them.
Yep :(
One of our requirements is that a pod lifetime is no longer than 2
weeks (we'll delete any pod older than 2 weeks, and the underlying
controller will recreate it). We also won't allow images with a
certain vulnerability rating to run, so you're pretty assured that
you're running current stuff. This might be a bit draconian for infra,
but something to consider/think about.
This sounds like a really good rule, how are you addressing if anything
gets broken during this recreation?
Similarly, python versions are all over the place... python3.6, 3.8,
3.9, etc. Of course different apps will target different versions
sometimes, but it might be nice to make sure we are using a supported
version by whatever container image the app is using. (ie, gets security
fixes).
See above, once you have unpatched vulnerabilities you can't run, so
this is effectively mitigated.
Anyhow, I'll stop there... but hopefully everyone would be ok with such
a doc and then changing things to match it?
Doc makes sense, and I might steal some of it :).
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
