Hello!
My apologies, I had accidentally dropped out of computer world for a while, and here a world spins ahead! Thanx a lot for many valuable feedback!
see the clarifications of most concerns:
> Might be a copy-paste error there, the last sentence is incomplete.
ty, fixed: "We already made a heavy testing of the behavior, and user should not face negative experience. Still I'm not sure if this testing can ever be enough, considering all the use-cases we do not know. "
> Bundled versions are always outdated and may be even vulnerable.
Not necessarily. In small project, sure, bundled libraries will get rotten, but project like OpenJDK, where 99% of its builds uses the in tree copies, can not allow itself to have security holes in them. It already happened in past that we
have switched to the in-tree library because the CVE there was already fixed, and moved back to system library once the fix landed to live Fedora.
> Using bundled freetype, fontconfig and harfbuzz will result in ugly fonts (without system configuration support, including subpixel rendering, hinting, etc).
This is probably main issue we are aware about. Especially the system configuration will be a hard issue, if solvable at all.
However IIRC, those features do nor work properly in java event hose days, as java have to support all what lies below, and it simply can not, so the intree libraries come to play.
Note that we had to patch jdk in past to work properly after changes in the system libraries. And that is not easy task.
Please take my last three sentences with bit of reserve, a better upstream engineer then me should comment.
> Applications linking against libraries SHOULD link against shared libraries not static versions. https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-stat...
right, should. So strictly from law of guidelines, there is no rule to stop this change. The bundling was always bad, but is necessary evil. See eg rsync - the bundled zlib is there because it is technically not possible to use dynamic one.
Java is in the border - yes, you can use the dynamic libraries (and it really took several excellent engineers a decade to manage), but it have pros and cons. Unluckily, the negatives are multiplying for years. Although we are not happy
with the change, it must be done if JDKs should remain maintained and healthy in Fedora.
s
> And upstream only incorporates security fixes once per quarter
Right, but downstream works well too. If such vulnerability occurs, be sure we will patch RPMs asap, and also upstream project - maybe without release - will react to.
Unluckily, I'm unable to comment on the demanglers, can you elaborate more?
> In this case upstream might actually get there first because this CVE is not yet fixed in Fedora
And it already happeed in past, that jdk swithced to bundled version for a while, and back to dynamic once the library was fixed in live Fedora.
> This sounds fascinating. Can anyone share details about this? On the
I'm aware of some codecs, which are built in Fedora, then the binary is sent to .. cisco(?), and if passed, they are repacked into all live fedoras.
As super cornercase of this may be packing of some firmwares as binary blobs.
> versions in an RPM sounds like it would violate all the guidelines under
> $foo to meet this threshold and claim it needs to avoid rebuilding for certification?
Right, you need fesco exception.
> I'm very confused on why this reduces certification burden. In our crypto libraries this is exactly the kind of behavior we would *NOT* want packages to do because it increases our certification and support burden.
I believe this are two things.
First - our burden. We ahve to certify each binary. This is quite long and lenghty process. Onl once it is certified, we can release it (with small unwritten exception in rawhide)
So with repacked portabel build as described in https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs will allow us to certify once (and even stop hoping for that small exception in rawhide). Considering 4 jdks in 3 maintianed
fedoras, that is incredible work. If it reduced to 4 we will manage again.
Second - I can understand that you certify your libray, but if others bundle it, then it is theirs choice, and they are on thier own, arent they? Also you highlight crypto lib, we are not going to embed any crypto library. Crypto is loaded
in runtime, and build time have no idea about future system in this case. Can you please elaborate more if still needed?
> I'm confused how this would not negatively impact the user experience, because things like FreeType and HarfBuzz in Fedora have features and configuration that are non-default that improve the font rendering capabilities of applications
that link to FreeType.
As explained above. This is knwon issue. I was naively considereding it as minor, as it nearly non-fixable, and to my half blind eyes some subixels and shadows says nothing. (/me on non patched fluxbox, which I guess explains all). This
is also how I wrote it to the https://fedoraproject.org/wiki/Changes/JdkInTreeLibsAndStdclibStatic#User_Experience and https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs#Side_effects . As I realy ahve only vague knowledge of
what you are speaking about, can you please ehnace those two paragraphs?
> I would rather have our shared maintenance and evolution of font stuff be reused in Java too...
This is holy grail we have been pursuing for last 10 years. But now we gave up. To keep java alive in Fedora, we have to take this one step back.
> Are people really installing OpenJDK RPM packages, taking the "/usr/bin/java" binary, and putting it onto some other system?
No, no and no:)
Please see the linked https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs
> I cannot see how Fedora RPM packages for OpenJDK can redistributing pre-built binaries would ever be considered acceptable.
no, no and no again, Please see the linked https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs
In long run, we will build in koji once, and repack this to all live fedoras. There alre already prcednets, and even worse things - like packing blobish binary driver, are here.
I guess you noticed, that it was clarified already in the discussion. Thanx a lot for highlighting this though.
> I agree, I don't think there's positives for the user experience here. And I don't understand what actual problem this change is trying to solve?
As described above, keeping dynamic jdk proeprly working requires huge maintainance. In addition, jdks are expected to be static (only distribution rpms do differently, and thats why developers often unpack 3rd party static jdk to develop
on), and have futures like generating standalone binary native images and so, which are impossible with dynamic linking. Future of java looks pointing pretty stright forward to such changes, so we have to move to.
> When the change talks about being "portable", my understanding is that means within the context of Fedora. eg they want to be able to build the JDK version in a Fedora 35 build root once, and then ship that built binary in Fedora 35,
Fedora 36 and Fedora rawhide (37) by just tagging the F35 build into the newer Fedora koji tags too, instead of rebuilding for each Fedora version.
Exactly. With one addition - the jdk will be indeed portable, you should be able to bring it to most of the system,and it should work. Still glibc limitations remains here. To stand on the safe ground, "usual" non distribution providers of
openjdk simply builds on oldest possible supported system.
Also there si one more advantage - the native images you produce form suchjdk will be trully portable. Unlike now, when they are simply unusable. I know distro-based people dont like this. But that si current state of world. By hart I'm
distribution person living on Fedora since Core 4 with few steps to other distros, but as java developer I;m rather distributing final self contianed blob to custommmer so he have everything in place. So despite being java maintainer for
decade, I' have to unpack and use 3rd party jdk much more often then I would like.
> This case appears different in that Fedora is still building the binaries in one release stream,
Right. Thanx for describing it.
> One downside with building in F35 and then re-tagging into newer Fedora releases, is that we loose any insight into problems coming down the pipe. Currently a Fedora rawhide mass rebuild will often highlight pre-existing bugs in
applications, and/or highlight bugs in newly rebased GCC toolchain....Detecting bugs early in both applications and GCC toolchain, via builds in rawhide...
Right you are, and it is already described in https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs#Rawhide_GCC
We are very well aware of this benefit, and we will do our best to keep it running - eg to build portbales also in rawhide, although not to repack them. Just for this single - but huge - case.
We deeply agree with your description of rawhide and gcc bumps. Thanx a lot for highligting it.
> This is so bad. The final death of Java on Fedora.
Actually just a opposite. This is future of java and without it java in fedora my diminish and fade away. I personally really do not like this change, and I agree with all rebukes taken here. But current OpenJDK maintainers (which are the
same for last decade) do not see other way.
If it will really go bad, we will withdraw and continue fighting.
> I consider this a step entirely in the wrong direction. The JDK should be linked to system libraries wherever possible just like our other packages. Language interpreters/JITs are not exempt from that. In fact, I see very little value
in providing JDK packages at all if they are built that way.
Where you are right with the interpreters/jits not being exempt, I'm afraid yo do not see current flow of java - to self contained native images, and to share OpenJDK marketpalce. Even if wesomhow will be able to pathc jdk to live with
system libraries, which is already close to impossible, and drop all features which allows generate portable images, the problem with the certification will persists. It isnot in human powers to maintain 4 JDKS in 3 Fedoras.
> And this plan is entirely unacceptable. It is just plain not allowed in Fedora to ship prebuilt binary blobs (even if they are built by Fedora developers)
This is wrong. The JDK will be always build from sources in koji. The main reduce of load will be that webuilt once in koji, in oldest Fedora, and repack to newer.
In additon there are many excludes in various binary drivers.
> If passing the TCK is such an issue, then please just go back to shipping the packages under the name IcedTea or some other name not trademarked by Oracle.
It is not as simple. You are right that to call it java yoou must pass tck. But unluckily even IcedTea had to pass TCK. Maybe it is possible to patch out all "java" from it, but I doubt jdk will remain working and usefull,.
> No way. This violates Fedora policy. All packages **must** be built from sources. No exceptions.
Quite a few packages are dleivered as blobs... Still. Be sure we are NOT going to do that. We will continue to build from sources, in koji, and with luck only repack build form latest stable Fedora to all,
> Fedora 38. So the decision on whether to approve this first feature really needs to consider the acceptability of the overall plan, not just this first step in isolation.
You are right, that the https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs is the key, but by aproving this, the whole chain isnotyet there. Especially if we found that the static jdks are really not viable, then this
(https://fedoraproject.org/wiki/Changes/JdkInTreeLibsAndStdclibStatic) proposal will be reverted, and the https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs cacneled.
But to be able to move forward, we have to try. If we dont, it will not know we are no wrong track, and thus be unabel to try to figure backup plan.
> IIUC, it is implied that when a new JDK comes out (jdk19 next), it would be added immediately to all Fedora streams (35, 36, 37rawhide).
Yes, jdk 19 will be added to all fedoras. But not as new package - it will replace jdk18 inside java-latest-openjdk. jdk19 is STS, simialrly as jdk18 was. Once next LTS is out jdk-21 in 2024, I hope we wil finally drop jdk8, so we wills
stay in 4 jdks.
> introduce new JDKs to all existing Fedora streams, only add it to rawhide so certification is only needed once.
That will unluckily not help. there would remain previous JDK in older Fedoras. In addition, EOLed one.
> certification; form same email as abvoe two huks, from Daniel P. Berrangé.
> + Why do we need such certification? Fedora is a separate distribution, not related to Oracle at all.
To call build of OpenJDK java, each binary ahve to pas TCK(as mentioned in thread). And you have to do it on each binary you produce. So with dynamic lining in fedoras, I have to certifie 12+ builds.
And update once they pass (with small exception in rawhide). With portable build, I can build once, TCK it eg on super custom opensuse, and still do update (from this binary) to all Fedoras.
TCK - are testing if all public apis forks as expected. Despite some tck are really weird, and to pass we eg have to have balck background and shadows disabled and so, the TCK are pretty good testsuite. Corenrcases are in other testssuites,
which we run too. As our build is based on OpenJDK with Hotspot, we do not need to prove it, but must provide once asked.
Also the "one would have thought that any time a dependency got an update it would invalidate certification too, even right down to any glibc update, or even kernel update " is close to reality. But we really have to pass the binary on ANY
system. Including the one before release. So luckily not.
> Is there an actual contractual requirement for Fedora to distribute OpenJDK builds only after they have passed the TCK? That's just impossible with the Fedora build system, and we would have to remove OpenJDK from Fedora to comply.
I'm not sure I follow. Why would we need to remove OpenJDK from fedora to comply? The only issue I see is the Rawhide. But considering it is not officially released.. It should be ok.
Each build we pass to updates, we are bound to prove it passes TCK. As we control environment, we are usually able to do so.
Runing them is mandatory. The results reporting is on demand. The simple rename is not so easy. I "m afraid that striping all "java" from bnaries and thus from sources is maybe possbel to be done, but human impossible to maintian, and
will leave JDK mostly not working, and for sure useless and incomaptible with other javas.
One of the side steps of this proposal is to be more compatible with other javas.
> If current maintainers can't continue maintaining the well-packaged OpenJDK, I think it's time to retire it. It would be better
Yes, we can no longer maintian it. And I must contradict you - Seeing all the dependencies, we really need them all. Loosing any one, wiould kill java ecosystemin Fedora.
> You do not need to rebuild OpenJDK after the system library update is installed.
We rebuild so often that this one is irrelevant.
> That would also mean that the JDKs would lag any other Fedora system-wide changes, such as compiler/library updates, build options that affect security, etc.
Well, yes. But except GCC which we will try to support as writen above, and in https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs#Rawhide_GCC
But please not, that that is very common troubles for us. We do adapt to fedora changes - eg crypto policies or so - but the bumps of libraries are unnecessary burden. As those usully need bigger patching upstream. But please takeme with
reserve - a more upstream/native libs development included persn should elaborate.
in last decade, we proved one thing - each fedora change like crypto policies or so, when finally tuned in JDK, was possible to backport to older fedoras. However changes due to shared libs, not.
> F35, F36, and rawhide need to be updated to a JDK built on F35... which would mean that F36 gets released with an immediately obsolete JDK, to be replaced with an untested (in the general sense, didn't go through OS beta and such) build.
Interesting point of view. Tyvm for it!
If the binary is identical, then what can go wrong? Already now tests portable comunity builds on all live fedoras, and a distribution-specific issue is so rare, that I think there was none even on long run.
Still I see yor point and possible issues you suggests out. Trhough tuccrent plan, your assumption "F36, and rawhide need to be updated to a JDK built on F35." is correct.
However the "t F36 gets released with an immediately obsolete JDK, to be replaced with an untested (in the general sense, didn't go through OS beta and such)" should not be correct.
There is ususaly no need to update immediately. It is actually even not desired. And here is few weeks where original builder Fedora is still alive, and we can stay on it for a while. But sooner or later the new "untested" from your point
of view Jdk will get there. The onl assurance it careful testing of Openjdk maintianers. To make your sleep a bit easier,:
gJobsAllOtool | grep -e f35 -e f34 -e f36 | wc -l
2736
We run 2736 testsuites jenkins jobs on Fedoras. From those,
gJobsAllOtool | grep -e f35 -e f34 -e f36 | grep tck | wc -l
197
197 are TCK (which each includes over 100k of tests) jenkins jobs, as there are several architectures, and even several variants of JDK which we wish to esnure. I expect the testing burden to drop to 1/2, without actually loosing necessary
coveragein matrix, after the whole https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs is alive.
> Or would rawhide get a build built on (oldest+1) rather than (oldest)? So F36 would be tested and released with a build built on ...
Interesting idea. Tyvm for it!
No, I was considering simply latest live one. But no imediate repack is needed once it is eoled. We can stay for a bit on eoled one, or not update from new oldest. In all cases the heavy testing will preceded
In your description. It is indeed a bit of confusing
>> JDKs from other vendors(Amazon, Azul, Oracle, etc.) are built in exactly this way. W
> Because they build it for all available GNU/Linux distributions. We shouldn't focus on bad practices.
Well and there are many reasons to do so, as jdk is designed to be like this. for 10 years we tried to go against the main stream, and we managed a lot. But to keep peace and compatibility and proper developer's support, we have to move
with mian stream now. The divergences we keep in rpms are right now blocking devloeprs to use system jdsk as proper JDKs and are enforcing them to download Amazon, Azul, Oracle, etc. blobs...
To keep Fedora competitive, this is currenlty necessary step.
I hope I addressed all,
Thanx a lot for contributions!
J.
--
Jiri Vanek Mgr.
Principal QA Software Engineer
Red Hat Inc.
+420 775 39 01 09
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
