On 13. 01. 19 11:41, Dan Horák wrote:
On Sun, 13 Jan 2019 11:05:33 +0100
Miro Hrončok <mhroncok@xxxxxxxxxx> wrote:
On 12. 01. 19 19:47, Kevin Fenzi wrote:
On 1/12/19 5:59 AM, Miro Hrončok wrote:
On 12. 01. 19 14:19, Kevin Kofler wrote:
Dan Horák wrote:
This is a reminder that this begins this Friday and will
probably be in place for 4 days. If there are increased
downtimes, we will update the data when we know it.
4 whole DAYS of Koji outage are absolutely unacceptable!
This just shows how bad an idea it was to add those exotic
secondary architectures to the primary Koji and to make them
blocking for all builds.
We really need to go back to building secondary architectures on
secondary
Koji instances, so that they do not hold the entire Fedora
hostage for days!
While I don't necessarily agree with the tone or wording, Kevin
has a point here.
I don't agree with the tone, wording or point. :)
Ah, it seems my e-mail was a little pointless without providing more
thoughts. Sorry about that.
When there were secondary arch koji's it took a number of people
full time to nurse along koji shadow. Often things would land in
primary, secondary would find a bug later and there would need to
be a lot of coordination and back and forth before things were
fixed in both places.
Now all those folks can concentrate on the bugs as they happen in
primary koji, fix them faster and everyone wins.
I agree with this. That clearly reduces a lot of work that would
otherwise be carried by a group of alternate arch fighters. Also, the
situation is more easily to read etc.
No doubt that having everything in one Koji is beneficial.
If there were constant outages you might have more of a point, but
this is the only multi-day one I can think of, it's over a weekend,
noarch packages can keep building and if there's a security or
urgent issue we can just Exclude s390x until it's back up.
It's however not just about outages. What bothers me about s390x is:
* From time to time there are situations where the build waits for
a s390x builder.
as they wait for other arch builders too
I guess what I wanted to say is that it's s390x more often than others.
I mean, I usually wait for arm as it takes long time to finish, and s390x
actually builds quite fast, however it is s390x that's often "blue" (no pun
intended).
* Upstreams don't have access to a s390x CI service and are often
unable to debug s390x problems without (very slow) virtualization.
we are aware of the problem, for all non-x86 arches, but there is no
simple solution
Good. I never suspected this would be easy.
* With ppc64 removal, s390x is the only Big Endian arch in the pool
(while ppc64 was easier to get to via CentOS CI).
I just think that we are burning a lot of resources without a clear
visible benefit. Don't get me wrong, I am no architectures expert and
I don't intent to pretend I am. It's just that all the s390x bugs
I've seen for packages I happen to help taking care of are from other
Fedora maintainers who fight FTBFS for their own packages etc., never
from any actual users.
s390x just feels to me like "that weird thing that often breaks and
nobody really cares about". I realize this view is very narrow and is
mostly based on feelings rather than facts, however it just how it
feels.
Hence I consider it unfortunate that s390x can block the whole
distro, even if it's just for a couple days.
is it really such a big problem, during a weekend?
It's not a big problem. I've never said it's big. I said I consider it
unfortunate that it can block the distro (aka not this particular outage but
rather the possibility). Chances are any arch can block the entire distro,
however I guess getting a spare s309x mainframe if something happens would be a
bit harder than getting builders for other arches.
But this is empty talk. All I wanted to say that "I consider it unfortunate",
**not** that "we must go and change it now" or something like that.
I'm not saying all this to start a flame or to blame one poor
architecture, I'd genuinely love to know the answers to:
* Where can upstreams get a nice CI as a service for s390x? [1]
nowhere yet, but how it is different from other non-x86 arches?
Upstream can run their CI on our infrastructure, a number of
them already does.
BTW is there any open-source "CI as a service" software that could be
just deployed (on any arch)?
I don't know. Does Jenkins work on any arch?
* What are the users of Linux on z and what packages/apps/tools are
they interested about? Do they run desktop environments, should I
e.g. bother fixing a bug in a desktop GUI application that controls a
3D printer? Or is it just OK to Exclude s390x on such apps? etc.
they probably won't control 3D printers, but the overall goal is to
make s390x looks as any other arch, thus we have for example
virtio-gpu, so for a user there won't be a difference when running
their desktop environment. In general having an heterogeneous
environment is useful because it increases the overall quality pointing
to buggy user code, toolchain bugs, etc.
We should try to fix everything for s309x, yet the primary motivation for that
is heterogeneous environment, not actual users using the stuff. I understand the
"overall quality" point, however I'm not sure if it always makes sense (aka if
it is always worth it).
* Where do we actually gain users of Fedora/s390x? When I search
for Linux s309x, I get to our Wiki (after several Ubuntu and IBM
links) [2], but that page is probably not very user targeted and
seems a bit outdated.
The users are primarily interested in the enterprise products for
s390x, but they are well aware that Fedora shows them what will appear
in the next version of enterprise products. Or they simply need to
run newer stuff than enterprise version provide.
I kinda suspected that :)
And yes, our wiki would appreciate more love :-)
CommOps are great help with this kind of stuff.
Thanks Dan for taking the time to answer my questions, even during a weekend ;)
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
