Neha, We with Josh testing mods related to this here https://github.com/ceph/ceph/pull/24938 And will do the same for mimic-x in preparation for nautilus, if sounds good? Thx YuriW On Mon, Nov 5, 2018 at 3:37 PM Neha Ojha <nojha@xxxxxxxxxx> wrote: > > Nathan, I don't think we want to revert it for 13.2.2. > > This is because the pg log hard limit feature currently doesn't seem > to work well in a partial upgrade, recovery/backfill scenario. So, > even if we do revert it in 13.2.3, this still leaves us a chance of > going into a split scenario, where some osds in the field, running > 13.2.2(with hard limit code) and others on 13.2.3(without the code), > may encounter http://tracker.ceph.com/issues/36686. > > Therefore, users who have succesfully upgraded to 13.2.2, shouldn't be > at any risk. > For users trying to upgrade to a version >= 13.2.2, I am going to make > a note of this issue and add the suggested workaround in Pending > Release Notes for mimic. > > Does that make sense? > > Thanks, > Neha > > On Mon, Nov 5, 2018 at 2:43 PM, Yuri Weinstein <yweinste@xxxxxxxxxx> wrote: > > Acknowledged > > > > On Mon, Nov 5, 2018 at 2:35 PM Nathan Cutler <ncutler@xxxxxxx> wrote: > >> > >> Thanks, Neha. The luminous revert was just merged and we'll cut 12.2.10 > >> to push it out to users. > >> > >> Regarding Mimic, will there be a revert there as well? Since the pg hard > >> limit patches are present in 13.2.2, it sounds like we'll need to revert > >> them before we release 13.2.3? > >> > >> (Note that Yuri was planning to start QE for 13.2.3 - Yuri, please hold > >> off on that for now?) > >> > >> Nathan > >> > >> On 11/5/18 6:50 PM, Neha Ojha wrote: > >> > Hi All, > >> > > >> > We have discovered an issue with the pg log hard limit > >> > patches(https://github.com/ceph/ceph/pull/23211, > >> > https://github.com/ceph/ceph/pull/24308), where a partial upgrade > >> > during backfill, can cause the osds on the previous version, to fail > >> > with "assert(trim_to <= info.last_complete)". Full description of the > >> > bug is here: http://tracker.ceph.com/issues/36686. > >> > > >> > These changes are in 13.2.2 and 12.2.9, and a workaround for users is > >> > to upgrade and restart all OSDs to a version with the pg hard limit, > >> > or only upgrade when all PGs are active+clean. > >> > > >> > Until we add capability to have the pg log hard limit work smoothly in > >> > the upgrade case, we will be reverting these changes, > >> > https://github.com/ceph/ceph/pull/24903, and releasing 12.2.10 as > >> > early as possible. > >> > > >> > We are also reverting https://github.com/ceph/ceph/pull/24902, which > >> > is a low impact bug, but might causes issues in the field. > >> > > >> > Sorry for any inconvenience caused due to this. > >> > > >> > Thanks, > >> > Neha > >> > > >> > >> -- > >> Nathan Cutler > >> Software Engineer Distributed Storage > >> SUSE LINUX, s.r.o. > >> Tel.: +420 284 084 037
