On Sat, 2024-11-30 at 17:27 +0000, Peter Robinson wrote: > On Thu, 28 Nov 2024 at 16:33, Adam Williamson > <adamwill@xxxxxxxxxxxxxxxxx> wrote: > > > > On Thu, 2024-11-28 at 09:26 +0000, Peter Robinson wrote: > > > On Thu, 28 Nov 2024 at 09:23, Adam Williamson > > > <adamwill@xxxxxxxxxxxxxxxxx> wrote: > > > > > > > > Hey folks! just wanted to alert you to an, ahem, situation today... > > > > > > > > I foolishly fiddled with the production greenwave config (I forgot > > > > staging greenwave was a thing...) and got burned. For a few minutes I > > > > pushed a bad config that disabled gating entirely, and since Bodhi > > > > checks whether Rawhide updates are ready to be pushed every two > > > > minutes, it cheerfully went right ahead and pushed every Rawhide update > > > > which was previously gated. > > > > > > Is that not the reason there's pull request and review for changes > > > before they're deployed? > > > > Yes, but because of Thanksgiving, nobody was around. I doubt this > > There's people in the EU with sysadmin-main that could have reviewed. > > > would've been found on review, anyway, because it involves jinja block > > newline handling, which is basically black magic, especially when mixed > > with ansible. Turns out something in that mix transparently enables the > > jinja feature called trim_blocks, which completely changes how they > > work. If you test the same template I wrote just by providing all the > > necessary variables and running `jinja2` on it, you get completely > > different output, so if I or anyone else had tested it that way, we > > would not have seen the problem that happened. Now I know about that I > > know how to test a template with trim_blocks, but I wouldn't have > > figured that out ahead of time. Of course, maybe somebody else already > > knew this :| > > > > What I really needed was to wait for someone to say hey, we should > > merge this then run it on staging first to see what happens :| Sorry > > about that. > > Maybe you would have got that with a PR to review ;-) To be clear, I did send a PR - https://pagure.io/fedora-infra/ansible/pull-request/2389 . But I merged it after a few hours because I wanted to clear all this template stuff out of my head and switch contexts. For whatever reason it never occurred to me that a possible risk was that gating would get effectively *completely disabled*. I was *expecting* it not to be right first time, but I somehow figured the worst that would happen would be that greenwave would just crash, or everything would *fail* gating. So that's why I figured it'd be okay to just go ahead and push it and fix up the errors once I knew what they were, or revert if I couldn't figure them out. Obviously I was wrong there :/ -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx https://www.happyassassin.net -- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
