Hi Tom, First, thank you for contacting with us, providing feedback and sharing ideas. I'm glad you wrote - it's quite uncommon for Java upstreams to see value in having their projects available in Fedora or other GNU/Linux distributions. On 11/11/2014 03:05 AM, Tom Dunstan wrote: > After a few emails going around a couple of weeks ago, I would like to > re-kick-off a discussion about how to get a functional Gradle build into > Fedora. Given the recent Fedora 21 beta, it also seems like a good time to > try to work out what might be achievable for Fedora 22. I think that packaging latest Gradle 2.x should be doable for Fedora 22. I will prepare an official Fedora change for Gadle in Fedora 22. We also considered packaging 1.x line (as compat package "gradle1"), but for now my focus was mainly on 2.1. (I will uptade to 2.2 soon.) > I've had a look at both the plan ( > https://fedoraproject.org/wiki/User:Mizdebsk/PackagingGradle) and Mikolaj's > work here https://github.com/mizdebsk/gradle-packaging, which look great, > and seems like a sane path to getting Gradle into Fedora. > > Feedback > --- > > Some feedback on the existing TODO list and work done so far: > > - TODO item "report usage of non-free JSON library (org.json:json)" - we > no longer use this The non-free JSON lib is a dependency of XStream. We have reported this as XSTR-763 [1]. This lib seems to be used during Gradle 2.1 build - building Gradle pulls it into cache at $HOME/.gradle. This lib is also hosted in Gradle repo [2]. Perhaps it should be explicitly excluded as XStream upstream claims it's a test dependency only? > - Porting to aether-ant-tasks - we would probably accept a pull request to > upstream this, as we'll need to do it at some point anyway. There may be > some work on our end to work out / around any differences between the > implementations. Has the Fedora Java team found much in the way of > incompatibility when porting between the two? I have the patch mostly ready. I am definitely going to open a pull request in future (I've already signed CLA), but the patch needs some polishing and extra testing first. There are two incompatibilities I discovered so far: Gradle exposes maven-model in API. maven-model 2.x and 3.x are mostly compatible, but not entirely, so I suppose that some third-party plugins or build scripts would be affected by this change. Second smaller issue: some internal fields of maven-ant-tasks (like authentication info) can be injected from DSL. I found out that one field was renamed from "userName" to "username". I think this should be solvable without breaking build scripts. > - Local mode / xmvn integration - this looks a lot cleaner than the gradle > 1.0 patches! I wonder if it would be possible to add the /usr/share/java > repo without patching gradle itself - maybe using a local init script to > add the repo when doing a Fedora build. But then the repo wouldn't be > available outside the Fedora build process. The utlimate goal is to have two installations of Gradle. First upstream-like, for standard use by Fedora users and second, with some necessary Fedora-specific extensions enabled. The second installation would have local repository (xmvn) enabled by default so that we don't have to patch build scripts of every software we want to build for Fedora using Gradle. > - Upgrading Jetty. Mostly we're pointing people to use the external Gretty > plugin. See below for concerns. AFAIK Gradle is at Java 1.5, but any Jetty version above 6 requires Java 1.6 or newer. Unfortunatelly this alone makes our Jetty patch not upstreamable :( > One concern that we have is that any user-visible changes made to Gradle > will result in non-portable build scripts. A user using e.g. Gradle 2.2 > that they installed using yum will end up using Jetty 9, which is very > different to the Jetty 6 that users of the standard Gradle distribution > will use. This may then cause issues for users using the standard > distribution to build the target project. We are aware of that. We'll try to minimize changes we apply to Gradle (possibly by pushing patches upstream), but in some cases this may not be possible. In case of Jetty 6, from technical PoV we could add it to Fedora, but it would be against our goals (for many reasons Fedora doesn't want to ship legacy, unmaintained software). Current solution (porting to Jetty 9) is definitely not ideal either and I'll try to think of some better way of solving the problem. > The same goes for any plugins > that might not be enabled (there were lots of these in the earlier > Fedora-Gradle 1.0 patches), The goal is to enable all stardard plugins. As of now all plugins are enabled. Only Scala Zinc compiler is disabled, but this should be eventually fixed. > or non-standard repositories such as xmvn. xmvn repository was added temporarly only, until this problem is solved the right way. (As I mentioned above, custom repository will be enabled only in local mode, used only during package build and activated by running Gradle with a different command, like "gradle-local"). > While we would obviously prefer perfect compatibility, I can certainly > understand not wanting to keep older versions of certain packages around > forever in Fedora. It would be nice if, when using different packages such > as Jetty, a warning were emitted in the build to notify the user that they > are using a different version to the normal Gradle distribution and that > the build may be non-portable. That's an interesting idea. We can add a warning. > Another thing to consider would be modifying the version string to include > "-fedora" or -"rpm" or something like that so that users with issues > turning up at forums.gradle.org can report when they're using the rpm-based > distribution. Is there a standard Fedora policy for doing such things? This is a good idea. For most packages we don't change upstream versions (but usually we don't patch code either, besides bugfixes). In some cases we add "Fedora" or "Red Hat" suffixes (for examples see "gcc --version" or "mvn -version" in RHEL). I already tried adding Fedora version suffix, but GradleVersion class implements strict version checking which rejected my first attempts to use customized version. But I'll find a way to do this properly. Do you have any recommendation how the version should look like to satisfy Gradle versioning rules? > Testing > --- > > We have a large set of tests that we run as part of our continuous > integration and release processes. These tests take some hours to complete. > It looks like at least some rpm specs in fedora have the ability to run > bundled tests. What is the policy for when these tests are run? Does the > fedora build machinery run full test suites even if they take that long? We > would really like to see the full set of tests run against the patched > Gradle to get some level of confidence that the patches applied have not > harmed build compatibility. During RPM package build we try to run all unit and integration tests included in project sources. All test dependencies must be packaged as Fedora package. If any test fails, the build is aborted and RPMs are not created until maintainer fixes or disables failing test. For external integration tests we have Taskotron [1]. It is still in development and not ready for general use, but once it's ready we can think of adding Gradle IT suite there. Test dependencies don't have to be packaged in Fedora. Tasktron tests are ran after in testing phase (after package is built, but before shipped) or on demand. Failing tests don't prevent package from being shipped. So far I haven't even been running Gradle unit tests during build. One reason for that is we don't have all test dependencies packaged and some tests would have to be ported to newer dependency versions. Some cases I ported code to different Maven or Jetty versions, but left tests unchanged. This is on my TODO list, but lower priority. Another reason is that I am experimenting a bit with different patches and I don't want to spend too much time on porting or fixing tests if the patch is going to be reverted or changed anyways. In short: with time we'll try to enable as many tests as possible, but for now we may skip tests to speed up development. > > Opportunities / Ideas > --- > One idea that came to mind was enhancing the application plugin to create > application distributions that have symlinks to the correct jar inside > /usr/share/java rather than copying the jar into the distribution - that > might make life easier for devs / packagers trying to package Gradle-built > java applications. That could be done. There is xmvn-subst utility, which can be used to replace JAR with a symlink, assuming that the JAR is coming from Fedora and has Maven metadata installed. > > Another idea to consider is allowing a user to add the Xmvn repo to their > buildscript / build classpaths without having to patch their project build > file. This could be achieved by dropping a file into $GRADLE_HOME/init.d > which adds the Xmvn repo when a particular system property is set. Another > system property could disable all other repos - this would be > building-rpm-mode. See > http://www.gradle.org/docs/current/userguide/init_scripts.html for an > example of how to do this. As explained above, this is the goal, but I don't know yet how it will be implemented. Most likely a separate Gradle home (sharing all or most of JARs by symlinking them), with some extensions added. Maven is a good example of such separation. We ship vanilla Maven (yes, all our patches have been upstreamed!) in /usr/share/maven. This is used when you invoke "mvn" command. But during package build "xmvn" command is invoked instead. It uses Maven home at /usr/share/xmvn, which is identical to Maven home except that it has Fedora-specific extensions enabled. > > Questions > --- > > Are you folks aware of particular packages that Gradle uses or other issues > that are likely to cause problems while iterating through phase 3 of the > bootstrap plan? In particular if there's anything that we can do to help > that process along, it would be good to know about it. Phase 3 is already done (I forgot to update the Wiki page). All we need to do before Gradle can be pushed to Fedora is polishing packages, reviewing them and fixing a few remaining bugs. > One that leaps to > mind is "shading" dependent libraries - we do this in a few places to stop > the api available to the buildscript from being polluted. Shading is indeed a big problem, in general. I believe that shading sould be done only as last resort and that there are better techniques (like class loaders) to prevent conflicts of different library versions or leaking of implementation classes through APIs. I have already unshaded Maven 3 JARs (this was possible after I ported Gradle not to use maven-ant-tasks). I will look into unshading all remaining libraries. A few random problems I found working on packaging Gradle (there are more, but this email is already getting very long...): * Ivy resolvers can't be used in Gradle 2.x any longer (and are deprecated in 1.x). There is no public API for defining custom repository. (My temporary solution: use internal repository APIs.) * There is no official way for providing Gradle extensions (by extension I mean addon that modifies Gradle behavior). Long term I think it would be great if Gradle could use some IoC thechniques to inject implementation classes, allowing extensions to easily override some standard Gradle components. (My temporary solution: patch Gradle code.) * Internal Gradle artifacts (such as core-impl) are not published in artifact repositories. This makes more difficult to write extensions that use internal APIs. (I assume this done intentionally to discourage from using Gradle internals. My solution: create my own repo with missing artifacts.) * gradle-launcher has "Class-Path" entry in manifest, which is not allowed in Fedora. I had to patch Gradle launchers to work without this entry, but I am affraid that there may be third-party launchers that assume Gradle can be ran with "java -jar gradle-launcher.jar", which won't work in Fedora. It would be nice if this could somehow be solved with upstream cooperation. > Although I guess it's a little bit off-topic for this list, do you know if > the Xmvn repo structure is compatible with debian's /usr/share/maven-repo? > > How different is the Xmvn repo from a normal maven or ivy repo on the > filesystem? This is pefectly fine for java-devel list. /usr/share/maven-repo/ in Debian is a standard Maven repository and Gradle should be able to use it out of the box (through Ivy, without any custom code). Fedora "repository" is not a typical artifact repository at all. It doesn't have a strict layout (artifacts can have arbitrary file names and be placed at arbitrary locations in the file system). XMvn knows where to find them by reading special metadada files stored in /usr/share/maven-metadata/. Besides that we use custom version resolution scheme, meaning that POMs from Fedora can't be directly used by Gradle or Maven as they are. Some artifacts don't have POMs at all - we have to generate effective POM from metadata on demand. Thanks, -- Mikolaj Izdebski Software Engineer, Red Hat IRC: mizdebsk [1] http://jira.codehaus.org/browse/XSTR-763 [2] https://repo.gradle.org/gradle/libs/org/json/json/20140107/json-20140107.pom -- java-devel mailing list java-devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/java-devel
