I would like to start here by saying that the ceph developers are doing a great job and that I'm grateful for that. This discussion is not about criticising any developers. It is about bringing a divergence in perception to the attention of everyone in the hope that the developers can do an even greater job and earn even more appreciation. Sometimes, discussions about a relationship status can hurt feelings, but no such hurting is intended. > Yeah, generally there is no much enthusiasm about supporting that among developers. I guess its because none of them is administrating any large production installation and personally experiencing all the shortcomings of the latest "stable releases". My enthusiasm changed from "wanting new features" at the beginning to "only move on when everything is really rock solid" after operating a medium-sized ceph cluster (now 1051 OSDs, 11.5PB raw) for a bit more than 3 years. My enthusiasm for upgrading is very very limited at the moment. With the current fast turn-over way forward, I observe more and more frequent serious issues along the upgrade path, for example, https://docs.ceph.com/en/latest/releases/pacific/#v16-2-6-pacific; and more and more issues left behind because of an apparent need/urge to move on. Passing a test suite on a short-lived test system is not the same thing as operating a peta-byte sized cluster that accumulates issues over time, sees effects of scale, has realistic long-term user load and suffers from rarely occurring critical bugs that are dragged on in the code base because they are only discovered after a release reached their end of life. I have seen too many discussions on the issue list and github, where my immediate impression was that "nobody here has ever seen a real ceph cluster in their life". The actual operators seem to have little influence on the decisions taken. I have never seen a post by the devs to the user list saying "we need to fix/change/whatever xyz and there are different options, which one do the soldiers on the ground prefer?" A really good example is this https://tracker.ceph.com/issues/48203. I'm pretty sure no operator of a real-size cluster would have recommended the path that was chosen. These things acummulate. So yeah, if the opinion is that the user-community is responsible for stable operations, I guess the user community needs to pull together and branch out an LTS version. As correctly implied by the response "... and longer-term support would require an infeasible amount of upgrade compatibility and testing" (or, as I would say, longer release cadence, less new features and more boring bug fixing), it will probably mean a full fork with all its consequences due to the large number of development releases during the life time of an LTS release. > First of all that's an open source - so developers tend to have higher influence to decision making. That is correct and one of the sources of joy in open source development. However, developers also like to have users. One of the consequences of a fork would be a loss users, so developers also tend to listen to users to avoid forks and paths split in extreme situations only. I think here my message as a user is that the "stable releases" are significantly less mature than the developers seem to believe and a 4 year support period that spans 2 major versions where the latest version is, well, usually not that stable, is actually not that long and by no means LTS. The actual implied upgrade period is every 2 years and every 4 years as an exception. For storage this is extremely fast, usual replacement cycles are 5-7 years as a minimum. > [...] The latest releases felt kind of rushed out rather than well tested. Exactly my gut feeling as well. When I say LTS, what I mean is "change the release cadence to 5 or 10 years" to avoid the need for rushing things out. Just make everything LTS in that sense and rock solid long-term stability a priority again. > ... among CLT ... CLT - Cross laminated timber :) Sorry, don't speak the acronym language. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Daniel Tönnißen <dt@xxxxxxx> Sent: 17 November 2021 14:40 To: Dan van der Ster Cc: Ceph Users Subject: Re: Why you might want packages not containers for Ceph deployments The demand for LTS - at least in our case - does not stem from unprofessionalism or biased opinion. It's the desire to stay up to date on security patches as much as possible while maintaining a well tested and stable environment. Both Pacific and Octopus (we’re currently on Nautilus) have some problems within themselves that made us not upgrading our cluster. The latest releases felt kind of rushed out rather than well tested. We (the LTS „faction“) still would like to keep up with security and minor patches on the cluster. We’re not talking about backporting new features. There’s a saying: „Never change a running system“ -- Mit freundlichen Grüßen aus Oberhausen Daniel Tönnißen (Systemadministrator) <https://www.kamp.de/> <https://www.kamp.de/unternehmen/iso-27001.html> KAMP Netzwerkdienste GmbH Vestische Str. 89−91 | 46117 Oberhausen Fon:+49 (0) 208.89 402-50 <tel:+492088940250> Fax: +49 (0) 208.89 402-40E-Mail:dt@xxxxxxx <mailto:dt@xxxxxxx> WWW: https://www.kamp.de Geschäftsführer: Heiner Lante | Michael Lante | Amtsgericht Duisburg | HRB Nr. 12154 | USt-IdNr.: DE120607556 HINWEIS: UNSERE HINWEISE ZUM UMGANG MIT PERSONENBEZOGENEN DATEN FINDEN SIE IN UNSERER DATENSCHUTZERKLÄRUNG UNTER HTTPS://WWW.KAMP.DE/DATENSCHUTZ.HTML <https://www.kamp.de/datenschutz.html> DIESE NACHRICHT IST NUR FÜR DEN ADRESSATEN BESTIMMT. ES IST NICHT ERLAUBT, DIESE NACHRICHT ZU KOPIEREN ODER DRITTEN ZUGÄNGLICH ZU MACHEN. SOLLTEN SIE IRRTÜMLICH DIESE NACHRICHT ERHALTEN HABEN, BITTE ICH UM IHRE MITTEILUNG PER E-MAIL ODER UNTER DER OBEN ANGEGEBENEN TELEFONNUMMER. > Am 17.11.2021 um 14:26 schrieb Dan van der Ster <dan@xxxxxxxxxxxxxx>: > > The CLT is discussing a more feasible alternative to LTS, namely to > publish an RC for each point release and involve the user community to > help test it. > This can be discussed at the user-dev meeting tomorrow. > https://pad.ceph.com/p/ceph-user-dev-monthly-minutes > (BTW I just restored that etherpad -- it had been spammed). > > Cheers, Dan > > > On Wed, Nov 17, 2021 at 2:10 PM Eneko Lacunza <elacunza@xxxxxxxxx> wrote: >> >> >> I think the desire for a LTS version comes from the perception that >> lately latest stable Ceph is not as stable as it has been before. >> >> So in that regard LTS version is a means, not the objective, at least >> for us. >> >> Cheers >> >> El 17/11/21 a las 14:01, Igor Fedotov escribió: >>> First of all that's an open source - so developers tend to have higher >>> influence to decision making. >>> >>> And you can replace "among developers" to "among CLT" in my previous >>> post... >>> >>> Hopefully this position can be shifter if there is a wide "feature >>> request" from the field hence please try to share your thoughts at the >>> upcoming meeting. >>> >>> >>> Igor >>> >>> On 11/17/2021 3:40 PM, Marc wrote: >>>> But since when do developers decide? Do you know any factory where >>>> factory workers decide what product they are going to make and not >>>> the product management??? IT is becoming such a refuge for undetected >>>> unprofessionals. >>>> >>>> >>>>> Yeah, generally there is no much enthusiasm about supporting that among >>>>> developers. But it would be nice to hear points from user side >>>>> anyway... >>>>> >>>>> Igor >>>>> >>>>> On 11/17/2021 2:29 PM, Peter Lieven wrote: >>>>>> Am 17.11.21 um 12:20 schrieb Igor Fedotov: >>>>>>> Hi Peter, >>>>>>> >>>>>>> sure, why not... >>>>>> See [1]. I read it that it is not wanted by upstream developers. If we >>>>> want it the community has to do it. >>>>>> >>>>>> Nevertheless, I have put it on the list. >>>>>> >>>>>> >>>>>> >>>>>> Peter >>>>>> >>>>>> >>>>>> [1] >>>>> https://lists.ceph.io/hyperkitty/list/dev@xxxxxxx/thread/J3M3JWER7DS4CM3 >>>>> >>>>> FNWLTG543X4VPJN7E/ >>>>>> >>>>> -- >>>>> Igor Fedotov >>>>> Ceph Lead Developer >>>>> >>>>> Looking for help with your Ceph cluster? Contact us at https://croit.io >>>>> >>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich >>>>> CEO: Martin Verges - VAT-ID: DE310638492 >>>>> Com. register: Amtsgericht Munich HRB 231263 >>>>> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >> >> Eneko Lacunza >> Zuzendari teknikoa | Director técnico >> Binovo IT Human Project >> >> Tel. +34 943 569 206 | https://www.binovo.es >> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun >> >> https://www.youtube.com/user/CANALBINOVO >> https://www.linkedin.com/company/37269706/ >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |