Hi everyone, Several things went wrong with the 12.2.6 release. Thank you to everyone (users and developers both) who help to quickly diagnose, understand, and fix the problems! Before we get caught up in everything else on our plates I'd like to make sure we don't loose track of what could have prevented the problems with that release. This is by no means a comprehensive list (and possibly not even accurate--please correct me if this doesn't seem right), but is hopefully a decent start. 1- A feature in master was backported, but a follow-on bug fix was missed. I'm sent a separate email about this one since I think it's a complicated tool/process issue. 2- The upgrade/luminous-p2p suite didn't run because of tracker #24760 (shaman/chacra/teuthology aren't retaining builds of tagged releases in a way that teuthology can consume). This hopefully would have caught crc errors (depends on the workload). Can we discuss this in infra standup (or whenever) Monday to come to a conensus on the right solution? 3- The upgrade/jewel-x test also missed the crc problem. Unclear exactly why. One theory is that it was masked by the osdmap crc failures currently in the suite (which we considered non-blocking). Another is that it missed the right workloads too (maybe needs ceph-fuse added in?). I opened #25057 and marked it high priority. 4- Once the release was out and known to be bad, we didn't have a way to revert the repo to prevent people from installing the bad version. Even if they were aware, in some cases (deb) there wasn't a way for them to explicitly get the older packages. (I think.) I think we should discuss this with #1 this week to brainstorm possible solutions. Perhaps something as simple as keeping full copies of the published repos for each release and linking to the latest/best one via symlinks? Thanks! sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
