On Wed, Nov 30, 2022 at 12:54 PM Peter Robinson <pbrobinson@xxxxxxxxx> wrote: > > > This is true, and probably also not "fixable". We need to make some > > amount of non-upstreamable patches to some crates (most notably, > > removing Windows- or mac OS-specific dependencies, because we don't > > want to package those), but in some cases, these are "incompatible" > > changes, and Rust *developers* should not be targeting our downstream > > sources that have these differences with actual upstream sources. > > Yet you say above "We *do* provide value to both users *and* > developers" yet you say developers shouldn't be targeting that work? We provide value to developers by basically running a huge free CI across a wide range of architectures. That doesn't mean that they should develop against crate sources we have in Fedora, similarly to how upstream Python developers probably should use pip/venv instead of "whatver is in Fedora". The same applies to basically every language ecosystem except C/C++, where the package manager story is just so bad that system libraries are actually the only reliable development environment. > > This is due to a limitation of how cargo handles target-specific > > dependencies - all dependencies that are *mentioned in any way* need > > to be *present* for it to resolve dependencies / enabled optional > > features / update its lockfile etc. But since we don't want to package > > bindings for Windows and mac OS system APIs, we need to actually patch > > them out, otherwise builds will fail. > > And that ends up being quite a bit of work from my point of view. Also > the way the packaging works with options things like devel or optional > features ends up being very painful. I will often drop out optional > features just so I can do less packaging. Recent versions of rust2rpm automatically generate a patch to remove non-linux dependencies, so that work has effectively been reduced to zero. Disabling unused optional dependencies can also be done with either a rust2rpm config option or with a simple patch, so that should not be a problem, either. I also don't understand why disabling some optional features is such a bad thing? I don't think we have a policy in Fedora that says we *must* enable *all possible functionality and optional features*. > > > We're doing okay with #1, but... I think #3 _even_ with all of the work in > > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game > > > engine and will probably have a few things it would be nice to package to > > > make available in Fedora Linux. I might not even mind maintaining Bevy > > > itself. > > > > Somebody actually already started packaging Bevy components - some > > packages are already approved and some are still pending review. Not > > sure what the progress has been there, but it's not *impossible*. > > Well nothing is *impossible* if you have enough stamina, resources or > whatever else. I don't find saying something isn't *impossible* > necessarily makes it compelling. I agree. And I think we can do better, as I said in my original post. > > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of > > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel, > > > which is *amazing*. But... that still is hundreds that I'd have to add. And > > > mostly they are things I don't know _anything_ about. > > > > You must realize that this is an extreme case. For many Rust > > applications that people want to package for Fedora, the number of > > dependencies that are missing is rather small, *because* most popular > > libraries are already packaged. > > In my experience, and I'm not packaging any form of gaming engine, > it's not an extreme case, for a few small things and one slightly > larger thing I *have* packaged I now maintain over 60 more packages. I > have another one I want to package and AFAICT I get to package another > 50-100 but frankly it's hard to tell, and this is something I have > control over and have been actively trying to do upstream reduction of > deps. As far as I know, salimma's intern worked on a tool to determine missing dependencies for Rust projects in Fedora recursively, which should make it easier to determine the exact set of things that you would need to do here. > And requests to make things like this easier have been open for over a year: > https://pagure.io/fedora-rust/rust2rpm/issue/140 Right. I'm sorry about the issue reports that have been open for a long time. rust2rpm / rust-packaging is an entirely community-run project. We don't get any support from Red Hat (or other places) for anything except the Rust compiler itself. And with the original developer of rust2rpm (Igor) being gone, nobody who understands the low-level stuff in there is still around. zbyszek and I are trying our best to keep things running, but at this point, we'll need basically a rewrite of both rust2rpm and the RPM dependency / provides generators to support new cargo features. And since I don't have much time for that right now (busy finishing my university degree), these things are still pending. > > Bevy is a bit special, because it (presumably) pulls in lots of GPU / > > OpenGL / Vulkan related libraries, which we didn't need to package for > > anything else yet, and it's also split into dozens of small libraries > > itself, which can be painful to package, that is true. > > > > We might need to reconsider how to package projects like this. I'm > > pretty sure we could find a way to package them in a way that's > > compatible with how we're currently doing things but would be much > > less busywork. > > Well if we're finding a solution for those sorts of projects we should > be doing it for all rust projects. A lot of projects have internal > virtual/workspace bits where they split up things up internally but > may have a dozen or sub projects. These make absolutely no sense to > split up into separate packages because they're all bundled together, > have to be bumped together yet as it standard at the moment you either > have to package them all separately or do nasty rpm spec hacking to > try and glue it into a single rpm. Again RFEs to make less busy work > have sat for well over a year, some even 3 years, with no progress: > > https://pagure.io/fedora-rust/rust2rpm/issue/141 Same here. I *would like to work on this and improve the situation for all of us (including me)*, but I just don't have the time to do that right now. > > Sure, but isn't that the case for most projects that a newcomer wants > > to package, regardless of programming language? Say, somebody wants to > > package some cool new Python project for machine learning, then > > there's probably also some linear algebra package or SIMD math library > > in the dependency tree that's missing from Fedora. How is that > > different? > > I think the big difference here from my experience, and I've packaged > right across the Fedora spectrum, is that most of the python style > projects are able to use a pretty comprehensive standard library and > crypto library which helps minimise a lot of the extras I've seen in > the rust ecosystem. The Rust standard library is intentionally narrow. I don't think that will change any time soon, so we'll have to deal with it. Either way, it looks like the ecosystem is starting to standardize around a few common libraries (serde for de/serialization, tokio as async runtime, hyper for HTTP clients + servers, etc.), so things might get less sprawling in these areas. > > Still, for large projects that are split into many crates like bevy, I > > think we could find a way to make packaging them a lot easier (i.e. > > not use one source package per crate, but use one source package for > > building a project that contains many crates). > > See my point above about workspaces. Glad that we're agreeing on something ;) Still, we need somebody to actually do the work and implement these features. Pity we don't know any big open-source-friendly companies that operate in this space ... > > This is *not* a waste of time. > > If we don't invest time to do that, many project's dependencies grow > > stale, and actually *increase* the need for us to maintain compat > > packages. > > Well it depends on your opinion of a waste of time. The whole need for > packaging and maintaining compat packages is a waste of time and this > point just proves the whole problem with busyness. I've wasted so much I don't understand this point. We *need* compat packages for certain cases, or we'd break people's packages, simple as that. I don't create compat packages if they're not needed? > time dealing with other people's projects just so I can update my own > packages to achieve something like a minor version bump for a bug fix > or CVE..... time that I personally no longer have to spare, which for > me and the time I have *now* to spend on Fedora I feel is not well > spent, I would sooner be spending my spare time doing Arm enablement > or adding features, not just basic package bumps with provide *none* > of that, or in some cases I have to wait a bunch of time for someone > (generally yourself - and yes I very much appreciate the work you do) > else to do the bumps because it's large and I don't have the in depth > rust knowledge to handle 100s of package updates. (snip) > > We only maintain compat packages where porting to the new version (and > > submitting the changes upstream) is not feasible. Again, isn't that > > how Fedora is supposed to work? > > To some degree, but when it blocks package updates or means someone > has to do 100s of PRs upstream to update one package it's not > something most people enjoy or have the time to do. But that's exactly my point? Am I speaking in tongues? If porting dependent packages to a new version of something is too much work, then keeping the old one around with a compat package will work just fine. > > Or even if making Rust crates first-class deliverables *did work*, it > > wouldn't give us the benefits of the current approach: > > - we ensure that all crates in Fedora *build* on all architectures > > Bundling would also ensure this too. Ensuring that they *build*, yes. With the caveat that figuring out where in your bundled dependencies a build error comes from is pretty tedious, and patching bundled dependencies is a bit of a hassle. > > - we ensure that most crates in Fedora pass their test suites on all > > architectures > > Again I don't see how bundling stops this. It just ... does? When using vendored dependencies, test suites of bundled crates *aren't run* by cargo at all, and there's also no way to do that, as far as I know. > > - we check all crates for objectionable content, licensing problems, etc. > > That's a requirement of packaging in Fedora anyway. True, but with most projects that bundle stuff, people just seem to assume things are "fine" and don't look too hard. I don't want to know that horrors lurk in vendor tarballs for the packages that use this approach ... (and this is part of the reason why NodeJS packaging in Fedora died). > > - we change build flags to default to dynamically linking to system > > libraries instead of statically linking against vendored copies > > I don't see what that value provides to end users. Do I really need to explain this point? I think linking against system OpenSSL is *way better* than statically linking to a random vendored copy of it. > > This would mean that we basically stop contributing things to the > > upstream Rust ecosystem: > > - we diagnose / report / fix architecture support issues > > I don't see how the bundling of things to build on all architectures, > vs the build it all as single packages changes this, if it fails to > build on Z-series either as a vendored bundle or not it would still be > discoverable and hence reportable/fixable by the Fedora community for > benefit of the wider community. Errors like that bubble to the top eventually, but are much harder to diagnose and fix in this environment than when you're dealing with a single library in isolation. > > - we port projects to new versions of dependencies > > That's valuable for other projects but not Fedora, and it's not > something I am going to do personally as a rust packager. If being an exclusive consumer of Rust projects instead of collaborating with upstream projects is your choice, then so be it. But that's not usually the way things are supposed to work in Linux distros, I think. > > I see this work in the upstream ecosystem as an important part of the > > work we do in packaging Rust crates for Fedora, > > and I would not want to endorse an approach that meant we no longer do > > these things. > > I don't actually see how a change stops this. But it would? If you could just "cargo vendor" and yeet that horror-tarball into dist-git, you just wouldn't know about vendored copy 1.2.3 of library foo being vulnerable to CVE X or having bug #1234, or just being outdated and no longer working on Linux 6 or something ... Most of these things *only* happen if you deal with libraries individually. > > > Rust packages include a lot of machine-readable metadata. We should be able > > > to watch for CVEs, RustSec, and other security notices even without encoding > > > the metadata in RPMs. License review could also be automated — the field in > > > Cargo.toml is supposed to be SPDX, so that's convenient. [3] > > > > I already monitor RustSec advisories and check *all of them* against > > Fedora packages. This takes up a miniscule amount of the time I spend > > on Rust packaging (because there's so few Rust security advisories). > > So if there's so few sec advisories, probably because of the style of > rust programming, again I don't see why even for single packages this > answers the question/statement Matthew made. It's just not worth it? I don't want to spend days of my time on implementing some automation for a task that takes me literally a few minutes every few months? > > This is already the case, though? > > Writing a spec file for a new crate is already automated to the point > > where "standard" crates can be 100% automatically generated and need > > zero manual edits. > > If manual changes *are* required, then these changes would also be > > required in the "first-class crate artifact" scenario, so you don't > > gain anything. > > And if there's other problems that are caught during package review, > > the distribution mechanism doesn't matter, either. > > I agree with most of this, but what is not automated is that I need to > submit and get reviewed dozens of packages vs just one and it's a > completely different work flow than what I need to do in c9s and RHEL. The way Rust is packaged in RHEL is an abomination, and most likely also not OK from a legal standpoint. That's the only thing I'll say to this ... > > In my experience, changing the distribution mechanism or packaging > > paradigm will often make things *worse* instead of better. For > > example, the implosion of the NodeJS package ecosystem in Fedora was > > not only caused by the horrid state NPM, but also because the new > > packaging guidelines which prefer bundling essentially made it > > impossible for packagers to verify that objectionable content is > > present in vendored dependencies. For Java, Modularity was seen as a > > "solution", but the result was that basically everybody - except for > > the Red Hat maintainers who maintained the modules - just stopped > > doing Java packaging because of the hostile environment. > > Frankly I don't think modularity solved any of those problems. I also > personally believed that most of the collapse is because RHEL doesn't > require separate packaging now and allows vendoring so there was > suddenly no need to package a million dependencies except for Fedora > so why bother? It saved a lot of enforced busyness and I can > completely see why that was done, because doing things like actual > feature development is more useful and the end users AKA customers > want actual features not perceived features like granular packaging. Frankly, I couldn't care less about RHEL, especially from a Rust packaging point of view. If they really cared about any of that, we'd get contributions to rust2rpm or RPM packaging tools from RH. But we don't. They just do their own franken-packaging by re-using some of the tools we developed in a way that they weren't intended to be used. > > Given that, I think we're actually in kind of a *good* situation with > > Rust packaging, especially compared to other language ecosystems - not > > only right now, but also looking at the future. And looking at the > > alternatives, all attempts at trying different approaches (maven > > artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.) > > have *failed* and ultimately made things worse instead of improving > > the situation - the only thing that has proven to be sustainable (for > > now) is ... maybe surprisingly, plain RPM packages. > > I don't think it's as bad as other ecosystems but I don't agree with > your assessment of "kind of a *good* situation", I feel there may be a > little bit of Stockholm syndrome coming into play here to be honest. Of course I'm invested in this stuff. I've been spending lots of my time on it. Of course it's not perfect. There's much space for improvements. The only thing we don't have is an abundance of people who work on it to actually make it better. > As someone who started rust packaging around 2.5 years ago I feel I > spent the vast majority of my time doing things that don't achieve > what I need to achieve from actually packaging those packages in the > first place but rather waste a lot of my time having to do lots and > lots of packaging and fixing things elsewhere so I can update in a lot > of cases is what is a single cli package AKA what you call a simple > rust package so I can get a bug fix or a new feature out to my users > and I don't think that is a good use of my time. Packages changing lots of their dependencies for a simple bugfix release seems like really bad upstream practice, and I don't think Rust is to blame here, but that's beside the point. Fabio _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
