Alfredo Deza <adeza@xxxxxxxxxx> writes: > I would like to raise once again the discussion of improving our > release process so that we can minimize these things happening at the > very last minute. > > This isn't the first time we've hit issues that had to be reverted > (luckily packages didn't get to download.ceph.com), and since we > insist on owning the package and repository creation, there > is no way we can retire a "bad" release that is available publicly. In this case, there wasn't a build kicked off yet, so there shouldn't be packages out yet, I agree improvements are needed. Maybe waiting for a day after Sage's approval allows for these things to be caught like in this release > A few of the items that I believe could improve are: > > * Start having stricter development and freeze periods > * Add a quiet period (or waiting time) of a week for a release to be public > * Determine what the SHA to release is, so that testing can be done > against that SHA So far QE is freezed on a SHA, and now repos are also locked down, but finding bugs outside of QE suite is usually one of the triggers that causes a QE validated test also going back for build again. As we improve our coverage hopefully this should die down (very optimistic forward looking statements) > * Avoid depending on back and forth emails asking leads if they are OK > with a release or if they need another PR to be merged - let's define > criteria to meet this. I agree this increases the lead time, but here an EAFP approach often leads to this critical fix was needed sort of a thing. Finalizing at the CLT might be easier given the final results aren't out on a thursday. > > On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@xxxxxxxxxx> wrote: >> >> https://github.com/ceph/ceph/pull/32018 has merged, we should be ready >> to build 14.2.5 now. >> >> On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@xxxxxxxxxx> wrote: >> > >> > David Zafman has discovered a buggy patch in nautilus, which we want >> > to revert before releasing 14.2.5. More details in >> > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The >> > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now. >> > We'll need to rebuild 14.2.5 once the revert merges. >> > >> > Sorry about the inconvenience. >> > >> > Neha >> > >> > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@xxxxxxxxxx> wrote: >> > > >> > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote: >> > > > Yuri Weinstein <yweinste@xxxxxxxxxx> writes: >> > > > >> > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests >> > > > > results, this is ready for publishing. >> > > > >> > > > Sage, is this ready to build and start publishing packages for? >> > > >> > > Yeah, I think we're good to go! >> > > >> > > Thanks everyone- >> > > sage >> > > >> > > > > Abhishek, Nathan FYI >> > > > > >> > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@xxxxxxxxxx> wrote: >> > > > >> >> > > > >> >> > > > >> >> > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@xxxxxxxxxx> wrote: >> > > > >>> >> > > > >>> Outstanding need approval: >> > > > >>> >> > > > >>> ceph-deploy - Sage >> > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing >> > > > >> >> > > > >> approved >> > > > >> >> > > > >>> >> > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing >> > > > >> >> > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048 >> > > > >> >> > > > >>> >> > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@xxxxxxxxxx> wrote: >> > > > >>> > >> > > > >>> > (This is an early update, some tests are still running, as we are >> > > > >>> > trying to release this point next week before the US holidays, and >> > > > >>> > have more time to review results) >> > > > >>> > >> > > > >>> > Details of this release summarized here: >> > > > >>> > https://tracker.ceph.com/issues/42839#note-3 >> > > > >>> > >> > > > >>> > rados - approved by Neha >> > > > >>> > rgw - approved by Casey >> > > > >>> > rbd - need approval Jason >> > > > >>> > krbd - need approval Jason, Ilya >> > > > >>> > fs - need approval Patrick, Ramana >> > > > >>> > kcephfs - need approval Patrick, Ramana >> > > > >>> > multimds - need approval Patrick, Ramana >> > > > >>> > ceph-deploy - FAILED Sage, Alfredo ? >> > > > >>> > ceph-disk - N/A >> > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A >> > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED >> > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED >> > > > >>> > upgrade/luminous-p2p - in progress >> > > > >>> > powercycle - in progress >> > > > >>> > ceph-ansible - Brad is finxing >> > > > >>> > upgrade/luminous-x (nautilus) - in progress >> > > > >>> > upgrade/mimic-x (nautilus) - in progress >> > > > >>> > ceph-volume - Jan fixing >> > > > >>> > (please speak up if something is missing) >> > > > >>> > >> > > > >>> > Thx >> > > > >>> > YuriW >> > > > >>> >> > > > > _______________________________________________ >> > > > > Dev mailing list -- dev@xxxxxxx >> > > > > To unsubscribe send an email to dev-leave@xxxxxxx >> > > > >> > > > -- >> > > > Abhishek Lekshmanan >> > > > SUSE Software Solutions Germany GmbH >> > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) >> > > > _______________________________________________ >> > > > Dev mailing list -- dev@xxxxxxx >> > > > To unsubscribe send an email to dev-leave@xxxxxxx >> > > > _______________________________________________ >> > > Dev mailing list -- dev@xxxxxxx >> > > To unsubscribe send an email to dev-leave@xxxxxxx >> > -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
