Thanks Guiilaume and Redo for tracking down this issue. After talking more with Guillaume I now realized that not all the tests were using the expected latest-reef-devel label so Rook tests were incorrectly showing green for Reef. :( Now that I ran the tests again in the test PR <https://github.com/rook/rook/pull/13203> with all tests using the latest-reef-devel label, all the tests with OSDs on PVCs are failing that use ceph-volume raw mode. So this is a blocker for Rook scenarios, we really need to fix to avoid breaking OSDs. Thanks, Travis On Wed, Nov 15, 2023 at 1:03 PM Guillaume Abrioux <gabrioux@xxxxxxx> wrote: > Hi Yuri, (thanks) > > Indeed, we had a regression in ceph-volume impacting rook scenarios which > was supposed to be fixed by [1]. > It turns out rook's CI didn't catch that fix wasn't enough for some reason > (I believe the CI run wasn't using the right image, Travis might confirm or > give more details). > Another patch [2] is needed in order to fix this regression. > > Let me know if more details are needed. > > Thanks, > > [1] > https://github.com/ceph/ceph/pull/54429/commits/ee26074a5e7e90b4026659bf3adb1bc973595e91 > [2] https://github.com/ceph/ceph/pull/54514/files > > > -- > Guillaume Abrioux > Software Engineer > > ------------------------------ > *From:* Yuri Weinstein <yweinste@xxxxxxxxxx> > *Sent:* 15 November 2023 20:23 > *To:* Nizamudeen A <nia@xxxxxxxxxx>; Guillaume Abrioux < > gabrioux@xxxxxxxxxx>; Travis Nielsen <tnielsen@xxxxxxxxxx> > *Cc:* Adam King <adking@xxxxxxxxxx>; Redouane Kachach <rkachach@xxxxxxxxxx>; > dev <dev@xxxxxxx>; ceph-users <ceph-users@xxxxxxx> > *Subject:* [EXTERNAL] Re: reef 18.2.1 QE Validation status > > This is on behalf of Guillaume. > > We have one more last mites issue that may have to be included > https://tracker.ceph.com/issues/63545 > https://github.com/ceph/ceph/pull/54514 > > Travis, Redo, Guillaume will provide more context and details. > > We are assessing the situation as 18.2.1 has been built and signed. > > On Tue, Nov 14, 2023 at 11:07 AM Yuri Weinstein <yweinste@xxxxxxxxxx> > wrote: > > > > OK thx! > > > > We have completed the approvals. > > > > On Tue, Nov 14, 2023 at 9:13 AM Nizamudeen A <nia@xxxxxxxxxx> wrote: > > > > > > dashboard approved. Failure known and unrelated! > > > > > > On Tue, Nov 14, 2023, 22:34 Adam King <adking@xxxxxxxxxx> wrote: > > >> > > >> orch approved. After reruns, orch/cephadm was just hitting two known > (nonblocker) issues and orch/rook teuthology suite is known to not be > functional currently. > > >> > > >> On Tue, Nov 14, 2023 at 10:33 AM Yuri Weinstein <yweinste@xxxxxxxxxx> > wrote: > > >>> > > >>> Build 4 with https://github.com/ceph/ceph/pull/54224 was built and > I > > >>> ran the tests below and asking for approvals: > > >>> > > >>> smoke - Laura > > >>> rados/mgr - PASSED > > >>> rados/dashboard - Nizamudeen > > >>> orch - Adam King > > >>> > > >>> See Build 4 runs - https://tracker.ceph.com/issues/63443#note-1 > > >>> > > >>> On Tue, Nov 14, 2023 at 12:21 AM Redouane Kachach < > rkachach@xxxxxxxxxx> wrote: > > >>> > > > >>> > Yes, cephadm has some tests for monitoring that should be enough > to ensure basic functionality is working properly. The rest of the changes > in the PR are for rook orchestrator. > > >>> > > > >>> > On Tue, Nov 14, 2023 at 5:04 AM Nizamudeen A <nia@xxxxxxxxxx> > wrote: > > >>> >> > > >>> >> dashboard changes are minimal and approved. and since the > dashboard change is related to the > > >>> >> monitoring stack (prometheus..) which is something not covered in > the dashboard test suites, I don't think running it is necessary. > > >>> >> But maybe the cephadm suite has some monitoring stack related > testings written? > > >>> >> > > >>> >> On Tue, Nov 14, 2023 at 1:10 AM Yuri Weinstein < > yweinste@xxxxxxxxxx> wrote: > > >>> >>> > > >>> >>> Ack Travis. > > >>> >>> > > >>> >>> Since it touches a dashboard, Nizam - please reply/approve. > > >>> >>> > > >>> >>> I assume that rados/dashboard tests will be sufficient, but > expecting > > >>> >>> your recommendations. > > >>> >>> > > >>> >>> This addition will make the final release likely to be pushed. > > >>> >>> > > >>> >>> On Mon, Nov 13, 2023 at 11:30 AM Travis Nielsen < > tnielsen@xxxxxxxxxx> wrote: > > >>> >>> > > > >>> >>> > I'd like to see these changes for much improved dashboard > integration with Rook. The changes are to the rook mgr orchestrator module, > and supporting test changes. Thus, this should be very low risk to the ceph > release. I don't know the details of the tautology suites, but I would > think suites involving the mgr modules would only be necessary. > > >>> >>> > > > >>> >>> > Travis > > >>> >>> > > > >>> >>> > On Mon, Nov 13, 2023 at 12:14 PM Yuri Weinstein < > yweinste@xxxxxxxxxx> wrote: > > >>> >>> >> > > >>> >>> >> Redouane > > >>> >>> >> > > >>> >>> >> What would be a sufficient level of testing (tautology > suite(s)) > > >>> >>> >> assuming this PR is approved to be added? > > >>> >>> >> > > >>> >>> >> On Mon, Nov 13, 2023 at 9:13 AM Redouane Kachach < > rkachach@xxxxxxxxxx> wrote: > > >>> >>> >> > > > >>> >>> >> > Hi Yuri, > > >>> >>> >> > > > >>> >>> >> > I've just backported to reef several fixes that I > introduced in the last months for the rook orchestrator. Most of them are > fixes for dashboard issues/crashes that only happen on Rook environments. > The PR [1] has all the changes and it was merged into reef this morning. We > really need these changes to be part of the next reef release as the > upcoming Rook stable version will be based on it. > > >>> >>> >> > > > >>> >>> >> > Please, can you include those changes in the upcoming reef > 18.2.1 release? > > >>> >>> >> > > > >>> >>> >> > [1] https://github.com/ceph/ceph/pull/54224 > > >>> >>> >> > > > >>> >>> >> > Thanks a lot, > > >>> >>> >> > Redouane. > > >>> >>> >> > > > >>> >>> >> > > > >>> >>> >> > On Mon, Nov 13, 2023 at 6:03 PM Yuri Weinstein < > yweinste@xxxxxxxxxx> wrote: > > >>> >>> >> >> > > >>> >>> >> >> ---------- Forwarded message --------- > > >>> >>> >> >> From: Venky Shankar <vshankar@xxxxxxxxxx> > > >>> >>> >> >> Date: Thu, Nov 9, 2023 at 11:52 PM > > >>> >>> >> >> Subject: Re: Re: reef 18.2.1 QE Validation > status > > >>> >>> >> >> To: Yuri Weinstein <yweinste@xxxxxxxxxx> > > >>> >>> >> >> Cc: dev <dev@xxxxxxx>, ceph-users <ceph-users@xxxxxxx> > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> Hi Yuri, > > >>> >>> >> >> > > >>> >>> >> >> On Fri, Nov 10, 2023 at 4:55 AM Yuri Weinstein < > yweinste@xxxxxxxxxx> wrote: > > >>> >>> >> >> > > > >>> >>> >> >> > I've updated all approvals and merged PRs in the tracker > and it looks > > >>> >>> >> >> > like we are ready for gibba, LRC upgrades pending > approval/update from > > >>> >>> >> >> > Venky. > > >>> >>> >> >> > > >>> >>> >> >> The smoke test failure is caused by missing (kclient) > patches in > > >>> >>> >> >> Ubuntu 20.04 that certain parts of the fs suite (via smoke > tests) rely > > >>> >>> >> >> on. More details here > > >>> >>> >> >> > > >>> >>> >> >> https://tracker.ceph.com/issues/63488#note-8 > > >>> >>> >> >> > > >>> >>> >> >> The kclient tests in smoke pass with other distro's and > the fs suite > > >>> >>> >> >> tests have been reviewed and look good. Run details are > here > > >>> >>> >> >> > > >>> >>> >> >> > https://tracker.ceph.com/projects/cephfs/wiki/Reef#07-Nov-2023 > > >>> >>> >> >> > > >>> >>> >> >> The smoke failure is noted as a known issue for now. > Consider this run > > >>> >>> >> >> as "fs approved". > > >>> >>> >> >> > > >>> >>> >> >> > > > >>> >>> >> >> > On Thu, Nov 9, 2023 at 1:31 PM Radoslaw Zarzynski < > rzarzyns@xxxxxxxxxx> wrote: > > >>> >>> >> >> > > > > >>> >>> >> >> > > rados approved! > > >>> >>> >> >> > > > > >>> >>> >> >> > > Details are here: > https://tracker.ceph.com/projects/rados/wiki/REEF#1821-Review . > > >>> >>> >> >> > > > > >>> >>> >> >> > > On Mon, Nov 6, 2023 at 10:33 PM Yuri Weinstein < > yweinste@xxxxxxxxxx> wrote: > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > Details of this release are summarized here: > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > https://tracker.ceph.com/issues/63443#note-1 > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > Seeking approvals/reviews for: > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > smoke - Laura, Radek, Prashant, Venky > (POOL_APP_NOT_ENABLE failures) > > >>> >>> >> >> > > > rados - Neha, Radek, Travis, Ernesto, Adam King > > >>> >>> >> >> > > > rgw - Casey > > >>> >>> >> >> > > > fs - Venky > > >>> >>> >> >> > > > orch - Adam King > > >>> >>> >> >> > > > rbd - Ilya > > >>> >>> >> >> > > > krbd - Ilya > > >>> >>> >> >> > > > upgrade/quincy-x (reef) - Laura PTL > > >>> >>> >> >> > > > powercycle - Brad > > >>> >>> >> >> > > > perf-basic - Laura, Prashant (POOL_APP_NOT_ENABLE > failures) > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > Please reply to this email with approval and/or > trackers of known > > >>> >>> >> >> > > > issues/PRs to address them. > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > TIA > > >>> >>> >> >> > > > YuriW > > >>> >>> >> >> > > > _______________________________________________ > > >>> >>> >> >> > > > Dev mailing list -- dev@xxxxxxx > > >>> >>> >> >> > > > To unsubscribe send an email to dev-leave@xxxxxxx > > >>> >>> >> >> > > > > > >>> >>> >> >> > > > > >>> >>> >> >> > _______________________________________________ > > >>> >>> >> >> > ceph-users mailing list -- ceph-users@xxxxxxx > > >>> >>> >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> > > >>> >>> >> >> -- > > >>> >>> >> >> Cheers, > > >>> >>> >> >> Venky > > >>> >>> >> >> > > >>> >>> >> _______________________________________________ > > >>> >>> >> ceph-users mailing list -- ceph-users@xxxxxxx > > >>> >>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >>> >>> _______________________________________________ > > >>> >>> ceph-users mailing list -- ceph-users@xxxxxxx > > >>> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >>> > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > Unless otherwise stated above: > > Compagnie IBM France > Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex > RCS Nanterre 552 118 465 > Forme Sociale : S.A.S. > Capital Social : 664 069 390,60 € > SIRET : 552 118 465 03644 - Code NAF 6203Z > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |