Clement brought up that spring cleaning of our ansible playbooks would be a good idea. This is painfully obvious during our previous update/reboot cycles where we have had services not updated or restarted correctly so that systems did not come up well when we rebooted. I have opened https://pagure.io/fedora-infrastructure/issue/7695 which is the tracking ticket for this problem. I am proposing that we do major updates something like the following in the future. We can tweak as we find better ways to do them in clusters later. If you maintain a service, take that playbook and add comments for the following: a. Who is the current maintainer b. Date when that was last updated c. Who tested the upgrade and when d. General comments to explain what things are doing. If the playbook should be retired, removed, killed, etc please do so. My goal will be to make our update schedules something like this: Day 1: a. Run update playbooks on staging instances. b. Fix any problems shown by those. c. Run general update vhost_update on staging instances d. Reboot staging instances. e. Fix problems found from this. Day 2: a. Access if day 1 was a complete failure and stop upgrade cycle b. Run update playbooks on low priority systems c. Fix any problems shown by those. d. Run general update vhost_update on staging instances e. Reboot staging instances. f. Fix problems found from this. Day 3: a. Access if day 2 was a complete failure and stop upgrade cycle b. Run update playbooks on high priority systems c. Fix any problems shown by those. d. Run general update vhost_update on staging instances e. Reboot staging instances. f. Fix problems found from this. This should cut down the extra long hours and extended outages we have needed to do in the last couple of reboot cycles. -- Stephen J Smoogen. _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
