Sage Weil <sweil@xxxxxxxxxx> writes:
> On Fri, 6 Dec 2019, Alfredo Deza wrote:
>> I would like to raise once again the discussion of improving our
>> release process so that we can minimize these things happening at the
>> very last minute.
>>
>> This isn't the first time we've hit issues that had to be reverted
>> (luckily packages didn't get to download.ceph.com), and since we
>> insist on owning the package and repository creation, there
>> is no way we can retire a "bad" release that is available publicly.
>
> Doesn't the fact that we own the distribution mean that we *can* yank a
> bad release? If we were relying on downstreams to do this for us we'd
> have no control at all. Currently the only reason we can't is because
> we're chafing against the tools.
>
>> A few of the items that I believe could improve are:
>>
>> * Start having stricter development and freeze periods
>> * Add a quiet period (or waiting time) of a week for a release to be public
>> * Determine what the SHA to release is, so that testing can be done
>> against that SHA
>> * Avoid depending on back and forth emails asking leads if they are OK
>> with a release or if they need another PR to be merged - let's define
>> criteria to meet this.
>
> I think the main missing piece here is deploying the release candidate
> into production before actually releasing. Most of these last minute
> issues were caught when we deployed on the lab cluster and my home
> cluster. I think this is similar to your 'quiet' period: once qa passes,
> we should deploy internally and make sure there aren't things the qa suite
> missed.
>
> The other problem was discovered in parallel by David and was totally
> unrelated to the release process/freeze/testing, but was deemed to be a
> release blocker. I don't think any process can avoid that.
>
> This release is a bit different, though, because there is a data
> corruption bug we really want to get out. Perhaps what we should have
> done here is a minimal release that only has that one bug fix so that our
> regression risk is basically 0 and we can expedite. Instead we grabbed
> everything that had already been tested and merged and went with that.
We currently merge to the release branch regularly after QE validation,
we could go to merges to {release}-next branches and only merge to
{release} after final QE validation and approval, this makes expedited
releases easier since we could merge a different branch containing only
fixes and merge that to the release branch, but workflow tools would need adoption
>
> sage
>
>
>>
>>
>> On Thu, Dec 5, 2019 at 1:09 PM Neha Ojha <nojha@xxxxxxxxxx> wrote:
>> >
>> > https://github.com/ceph/ceph/pull/32018 has merged, we should be ready
>> > to build 14.2.5 now.
>> >
>> > On Wed, Dec 4, 2019 at 5:02 PM Neha Ojha <nojha@xxxxxxxxxx> wrote:
>> > >
>> > > David Zafman has discovered a buggy patch in nautilus, which we want
>> > > to revert before releasing 14.2.5. More details in
>> > > https://github.com/ceph/ceph/pull/31970#issuecomment-561913632. The
>> > > revert PR https://github.com/ceph/ceph/pull/32018 is being tested now.
>> > > We'll need to rebuild 14.2.5 once the revert merges.
>> > >
>> > > Sorry about the inconvenience.
>> > >
>> > > Neha
>> > >
>> > > On Wed, Dec 4, 2019 at 11:28 AM Sage Weil <sweil@xxxxxxxxxx> wrote:
>> > > >
>> > > > On Wed, 4 Dec 2019, Abhishek Lekshmanan wrote:
>> > > > > Yuri Weinstein <yweinste@xxxxxxxxxx> writes:
>> > > > >
>> > > > > > David, assuming Sage is OK with `ceph-deploy` and overall tests
>> > > > > > results, this is ready for publishing.
>> > > > >
>> > > > > Sage, is this ready to build and start publishing packages for?
>> > > >
>> > > > Yeah, I think we're good to go!
>> > > >
>> > > > Thanks everyone-
>> > > > sage
>> > > >
>> > > > > > Abhishek, Nathan FYI
>> > > > > >
>> > > > > > On Wed, Nov 27, 2019 at 12:38 PM Neha Ojha <nojha@xxxxxxxxxx> wrote:
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> On Wed, Nov 27, 2019 at 8:01 AM Yuri Weinstein <yweinste@xxxxxxxxxx> wrote:
>> > > > > >>>
>> > > > > >>> Outstanding need approval:
>> > > > > >>>
>> > > > > >>> ceph-deploy - Sage
>> > > > > >>> upgrade/luminous-x (nautilus) - Neha, Josh reviewing
>> > > > > >>
>> > > > > >> approved
>> > > > > >>
>> > > > > >>>
>> > > > > >>> upgrade/mimic-x (nautilus) - Neha, Josh reviewing
>> > > > > >>
>> > > > > >> approved, failure tracked in https://tracker.ceph.com/issues/43048
>> > > > > >>
>> > > > > >>>
>> > > > > >>> On Fri, Nov 22, 2019 at 7:22 AM Yuri Weinstein <yweinste@xxxxxxxxxx> wrote:
>> > > > > >>> >
>> > > > > >>> > (This is an early update, some tests are still running, as we are
>> > > > > >>> > trying to release this point next week before the US holidays, and
>> > > > > >>> > have more time to review results)
>> > > > > >>> >
>> > > > > >>> > Details of this release summarized here:
>> > > > > >>> > https://tracker.ceph.com/issues/42839#note-3
>> > > > > >>> >
>> > > > > >>> > rados - approved by Neha
>> > > > > >>> > rgw - approved by Casey
>> > > > > >>> > rbd - need approval Jason
>> > > > > >>> > krbd - need approval Jason, Ilya
>> > > > > >>> > fs - need approval Patrick, Ramana
>> > > > > >>> > kcephfs - need approval Patrick, Ramana
>> > > > > >>> > multimds - need approval Patrick, Ramana
>> > > > > >>> > ceph-deploy - FAILED Sage, Alfredo ?
>> > > > > >>> > ceph-disk - N/A
>> > > > > >>> > upgrade/client-upgrade-hammer (nautilus) - N/A
>> > > > > >>> > upgrade/client-upgrade-jewel (nautilus) - PASSED
>> > > > > >>> > upgrade/client-upgrade-mimic (nautilus) - FAILED
>> > > > > >>> > upgrade/luminous-p2p - in progress
>> > > > > >>> > powercycle - in progress
>> > > > > >>> > ceph-ansible - Brad is finxing
>> > > > > >>> > upgrade/luminous-x (nautilus) - in progress
>> > > > > >>> > upgrade/mimic-x (nautilus) - in progress
>> > > > > >>> > ceph-volume - Jan fixing
>> > > > > >>> > (please speak up if something is missing)
>> > > > > >>> >
>> > > > > >>> > Thx
>> > > > > >>> > YuriW
>> > > > > >>>
>> > > > > > _______________________________________________
>> > > > > > Dev mailing list -- dev@xxxxxxx
>> > > > > > To unsubscribe send an email to dev-leave@xxxxxxx
>> > > > >
>> > > > > --
>> > > > > Abhishek Lekshmanan
>> > > > > SUSE Software Solutions Germany GmbH
>> > > > > GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
>> > > > > _______________________________________________
>> > > > > Dev mailing list -- dev@xxxxxxx
>> > > > > To unsubscribe send an email to dev-leave@xxxxxxx
>> > > > > _______________________________________________
>> > > > Dev mailing list -- dev@xxxxxxx
>> > > > To unsubscribe send an email to dev-leave@xxxxxxx
>> >
>>
>>
--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg)
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
