People Involved
- Nigel
- Amar
Timeline of events (in IST)
1725 - Nigel merges Amar’s patch with the .clang-format file
1727 - Nigel lands the .clang-format changes to master as gluster-ant. Smoke jobs pass at this point)
1746 - Amar notices that some files are missing in the clang patch.
1752 - Nigel lands a new patch with the missing files (all .c files)
1811 - Amar notices compilation issues after landing the .c changes because it modifes files with the pattern -tmpl.c. Amar starts working on a fix.
1839 - Nigel notices that the Jenkins job for clang-format doesn’t fail when it’s supposed to fail and goes to fix.
1855 - Clang-format Jenkins job is now fixed.
1906 - Amar’s fixes are merged with manual votes for Smoke and Centos Regression from the Infra team. At this point, the builds were passing, but we had voting issues
What Went Wrong
- We staged the changes on Github on Sept 10th. Given the size of changes, we we missed that the command used to make the changes only caught
.hfiles and not.cfiles. The following is the command in question.find . -path ./contrib -prune -o -name '*.c' -or -name '*.h' -print | xargs clang-format -i - With the changes that we landed, we did run into build bugs 1, 2 and fixed them. However, we did not verify that all the files were in fact modified or sync up on the find command.
- We had a general framework of agreement on the steps to do but we looked at it as a code change rather than an infrastructure change. There wasn’t a well defined go/no-go checklist.
- In the middle of this, we had a freebsd-builder enabled that made the smoke job for the final fix not vote. This needed a manual vote from the Infra team.
What Went Well
- We did reasonably good planning to find potential issues and in fact, did find some potential issues.
- Nigel and Amar were on hand and available to fix any potential issues that popped up
- The changes landed at the end of a working day for India the day before a public holiday. While there was impact, it was much less than a similar change performed at working hours.
Future recommendations
- Template files need to be caught by the clang-format job correctly so that they are not checked for formatting. Or the name of the file needs to be changed so that they don’t end with
.cor.h. - In the future, high impact changes need a good process which has at least an acceptance criteria, go/no-go checklist, and a rollback procedure.
- This work is currently incomplete and the bug3 tracks the remaining action items.
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
