A re-roll of the series to have "make coccicheck" run incrementally. For the v1 see: https://lore.kernel.org/git/cover-0.5-00000000000-20220825T141212Z-avarab@xxxxxxxxx/ Changes since v1 (based on SZEDER's feedback, thanks!): * Corrected small bits of commit message, removed mention of a "bug" (which was really something in an earlier series of mine...) * Split up the previous "incremental" commit. Now 1/9 is an unrelated fix that was in it (but which was spotted by it). * Added some preceding patches to make tweaking spatch flags easier. * The part where we optionally piggy-back on the dependencies of *.o files is now in 8/9, the "pure" incremental coccicheck in 7/9 will never need a compiler. * The "make coccicheck" now acts as before, i.e. no "make -k" needed, it runs to completion. I think having it "make -k" friendly would optionally make sense, but that can be left for some other time. * The previous change dropped the cocci *.log, we now log the stderr we had as before. * The spatchcache contrib command in 9/9 has updated docs & behavior. * The spatchcache now requires SPATCH_FLAGS=--very-quiet, which was the new default in this series before. I.e. we never cache things that have any stderr output. Ævar Arnfjörð Bjarmason (9): cocci rules: remove unused "F" metavariable from pending rule Makefile: add ability to TAB-complete cocci *.patch rules Makefile: have "coccicheck" re-run if flags change Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading cocci: split off include-less "tests" from SPATCH_FLAGS cocci: split off "--all-includes" from SPATCH_FLAGS cocci: make "coccicheck" rule incremental cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES spatchcache: add a ccache-alike for "spatch" .gitignore | 1 + Makefile | 113 +++++++-- contrib/coccinelle/spatchcache | 234 ++++++++++++++++++ .../coccinelle/the_repository.pending.cocci | 1 - shared.mak | 3 +- 5 files changed, 325 insertions(+), 27 deletions(-) create mode 100755 contrib/coccinelle/spatchcache Range-diff against v1: -: ----------- > 1: 72b6a8e4e0b cocci rules: remove unused "F" metavariable from pending rule 1: 8604ac9bfbc = 2: 0998948b881 Makefile: add ability to TAB-complete cocci *.patch rules 2: db3b3427668 ! 3: 63cf9f58d99 Makefile: have "coccicheck" re-run if flags change @@ Commit message since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a - "make clean". + "make cocciclean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated -: ----------- > 4: 54d6bae3984 Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading -: ----------- > 5: 4f165bf6128 cocci: split off include-less "tests" from SPATCH_FLAGS -: ----------- > 6: c74d09f4825 cocci: split off "--all-includes" from SPATCH_FLAGS 3: 6fa83695f1f ! 7: 120607b5da6 cocci: make "coccicheck" rule incremental @@ Commit message cocci: make "coccicheck" rule incremental Optimize the very slow "coccicheck" target to take advantage of - incremental rebuilding, and fix outstanding bugs and dependency - problems with the existing rule. + incremental rebuilding, and fix outstanding dependency problems with + the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of - make's parallelism combined with $(SPATCH_BATCH_SIZE) and "xargs -n". + make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the + "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. @@ Commit message make -W grep.c coccicheck - Would be the same as doing a full re-run. A change in a single file - would force us to do a full re-run. + Would do a full re-run, i.e. a a change in a single file would force + us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to - pick up where we left off. Note that (per [1]) we've had a race - condition since 960154b9c17 (coccicheck: optionally batch spatch - invocations, 2019-05-06) which might result in us producing corrupt - patches to to the concurrent appending to "$@+" in the pre-image. - - That bug is now fixed. + pick up where we left off, or how to incrementally merge e.g. a + "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was @@ Commit message file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the <RULE>/<FILE> files for a given <RULE>. + In the case discussed above of "grep.c" being changed we'll do the + full "cat" every time, so they resulting *.cocci.patch will always + be correct and up-to-date, even if it's "incrementally updated". + See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. - * We depend on a .build/<RULE>.ok file, which in turn depends on our - <RULE>. This emulates the previous "test -s" test, but in a more - Makefile-native way (e.g. -k works as expected). - - As noted above this change fixes existing dependency issues, - e.g. changing strbuf.h will now trigger a re-build. Our default is to - depend on $(FOUND_H_SOURCES). This makes incremental re-building - possible, but it's still fairly slow, as modifying one header will - cause a full re-build of everything. - - Which is why we'll not depend on $(FOUND_H_SOURCES) but the *.o file - corresponding to the *.c file, if it exists already. This means that - we can do: - - make all - make coccicheck - make -W column.h coccicheck - - By depending on the *.o we piggy-back on - COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the - "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that - for the *.sp files. E.g.: - - make contrib/coccinelle/free.cocci.patch - make -W column.h contrib/coccinelle/free.cocci.patch - - Will take around 15 seconds for the second command on my 8 core box if - I didn't run "make" beforehand to create the *.o files. But around 2 - seconds if I did and we have those "*.o" files. - - Notes about the approach of piggy-backing on *.o for dependencies: - - * It *is* a trade-off since we'll pay the extra cost of running the C - compiler, but we're probably doing that anyway. The compiler is much - faster than "spatch", so even though we need to re-compile the *.o to - create the dependency info for the *.c for "spatch" it's - faster (especially if using "ccache"). + As before we'll: - * If that *.o file doesn't exist we'll depend on an intermediate file - of ours which in turn depends on $(FOUND_H_SOURCES). + * End up generating a contrib/coccinelle/swap.cocci.patch, if we + "fail" by creating a non-empty patch we'll still exit with a zero + exit code. - This covers both an initial build, or where "coccicheck" is run - without running "all" beforehand, and because we run "coccicheck" - on e.g. files in compat/* that we don't know how to build unless - the requisite flag was provided to the Makefile. - - Most of the runtime of "incremental" runs is now spent on various - compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and - therefore conflate whether we *can* compile an object and generate - dependency information for it with whether we'd like to link it - into our binary. - - Before this change the distinction didn't matter, but now one way - to make this even faster on incremental builds would be to peel - those concerns apart so that we can see that e.g. compat/mmap.c - doesn't depend on column.h. + Arguably we should move to a more Makefile-native way of doing + this, i.e. fail early, and if we want all of the "failed" changes + we can use "make -k", but as the current + "ci/run-static-analysis.sh" expects us to behave this way let's + keep the existing behavior of exhaustively discovering all cocci + changes, and only failing if spatch itself errors out. Further implementation details & notes: - * We can take better advantage of parallelism, while making sure that - we don't racily append to the contrib/coccinelle/swap.cocci.patch - file from multiple workers. - - Before this change running "make coccicheck" would by default end + * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. @@ Commit message setting that makes sense when doing a non-incremental run of "make coccicheck". - * The change here to - "contrib/coccinelle/the_repository.pending.cocci" is because we've - been spewing out this warning into our $@.log since that rule was - added: - - warning: rule starting on line 21: metavariable F not used in the - or context code - - It's being fixed here because we'll now notice on "make - coccicheck-pending", since we're not redirecting all of STDERR to a - log file, instead we use the "--very-quiet" option. - Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". - As noted upthread of [2] a naïve removal of "--all-includes" will + As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/<file>.o.d and supply them with the "--include" option to spatch(1).q - * We can generalize COMPUTE_HEADER_DEPENDENCIES to spew out the list - of *.h dependencies without making a *.o object, which is slightly - faster, and would also benefit the "make sparse" target. We'd still - need to run the compiler, it would just have less work to do. - - 1. https://lore.kernel.org/git/877dlwotjc.fsf@xxxxxxxxxxxxxxxxxxx/ - 2. https://lore.kernel.org/git/87ft18tcog.fsf@xxxxxxxxxxxxxxxxxxx/ + 1. https://lore.kernel.org/git/87ft18tcog.fsf@xxxxxxxxxxxxxxxxxxx/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> ## Makefile ## -@@ Makefile: SP_EXTRA_FLAGS = -Wno-universal-initializer - SANITIZE_LEAK = - SANITIZE_ADDRESS = - --# For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will +@@ Makefile: SANITIZE_ADDRESS = + SPATCH_INCLUDE_FLAGS = --all-includes + SPATCH_FLAGS = + SPATCH_TEST_FLAGS = +-# Setting SPATCH_BATCH_SIZE higher will -# usually result in less CPU usage at the cost of higher peak memory. -# Setting it to 0 will feed all files in a single spatch invocation. - SPATCH_FLAGS = --all-includes -SPATCH_BATCH_SIZE = 1 # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change TRACK_SPATCH_DEFINES = - TRACK_SPATCH_DEFINES += $(SPATCH) +@@ Makefile: TRACK_SPATCH_DEFINES += $(SPATCH) + TRACK_SPATCH_DEFINES += $(SPATCH_INCLUDE_FLAGS) TRACK_SPATCH_DEFINES += $(SPATCH_FLAGS) + TRACK_SPATCH_DEFINES += $(SPATCH_TEST_FLAGS) -TRACK_SPATCH_DEFINES += $(SPATCH_BATCH_SIZE) GIT-SPATCH-DEFINES: FORCE @FLAGS='$(TRACK_SPATCH_DEFINES)'; \ @@ Makefile: check: $(GENERATED_H) COCCI = $(wildcard contrib/coccinelle/*.cocci) +COCCI_NAMES = $(COCCI:contrib/coccinelle/%.cocci=%) COCCI_PATCHES = $(addsuffix .patch,$(COCCI)) -+COCCI_PATCHES_OK = $(COCCI_PATCHES:%=.build/%.ok) COCCICHECK_PENDING = $(filter %.pending.cocci.patch,$(COCCI_PATCHES)) -COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) +COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_PATCHES)) -+COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok) -+COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok) COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) -$(COCCI_PATCHES): GIT-SPATCH-DEFINES @@ Makefile: check: $(GENERATED_H) - fi; \ - if ! echo $(COCCI_SOURCES) | xargs $$limit \ - $(SPATCH) $(SPATCH_FLAGS) \ +- $(SPATCH_INCLUDE_FLAGS) \ - --sp-file $< --patch . \ - >$@+ 2>$@.log; \ +.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) @@ Makefile: check: $(GENERATED_H) +define cocci-rule + +## Rule for .build/$(1).patch/$(2); Params: -+# 1 = $(1) -+# 2 = $(2) ++# $(1) = e.g. "free.cocci" ++# $(2) = e.g. "grep.c" +COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) +.build/$(1).patch/$(2): GIT-SPATCH-DEFINES -+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) ++.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES +.build/$(1).patch/$(2): $(1) +.build/$(1).patch/$(2): .build/$(1).patch/% : % + $$(call mkdir_p_parent_template) -+ $$(QUIET_SPATCH)$$(SPATCH) $$(SPATCH_FLAGS) \ -+ --very-quiet \ -+ --sp-file $(1) --patch . \ -+ $$(SPATCH_PATCH_FLAGS) $$< >$$@ ++ $$(QUIET_SPATCH)if ! $$(SPATCH) $$(SPATCH_FLAGS) \ ++ $$(SPATCH_INCLUDE_FLAGS) \ ++ --sp-file $(1) --patch . $$< \ ++ >$$@ 2>$$@.log; \ + then \ +- cat $@.log; \ ++ echo "ERROR when applying '$(1)' to '$$<'; '$$@.log' follows:"; \ ++ cat $$@.log; \ + exit 1; \ +- fi; \ +- mv $@+ $@; \ +- if test -s $@; \ ++ fi +endef + +define cocci-matrix-2 + -+$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) ++$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s))) +endef +define cocci-matrix-1 +$(foreach c,$(COCCI),$(call cocci-matrix-2,$(c))) @@ Makefile: check: $(GENERATED_H) +define spatch-rule + +contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) -+ $(QUIET_SPATCH_M)cat $$^ >$$@ -+endef -+ -+$(COCCI_PATCHES_OK): .build/%.ok : % -+ $(QUIET_SPATCH_R)if test -s $<; \ ++ $(QUIET_SPATCH_M)cat $$^ >$$@ && \ ++ if test -s $$@; \ then \ -- cat $@.log; \ -+ cat $< | sed -e 's/^/ /'; \ - exit 1; \ -- fi; \ -- mv $@+ $@; \ -- if test -s $@; \ -- then \ - echo ' ' SPATCH result: $@; \ -- fi -+ fi && \ -+ >$@ ++ echo ' ' SPATCH result: $$@; \ + fi ++endef + +ifneq ($(filter $(COCCI_RULES),$(MAKECMDGOALS)),) +$(eval $(foreach n,$(COCCI_NAMES),$(call spatch-rule,$(n)))) @@ Makefile: check: $(GENERATED_H) COCCI_TEST_RES_GEN = $(addprefix .build/,$(COCCI_TEST_RES)) $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES -@@ Makefile: $(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinell - coccicheck-test: $(COCCI_TEST_RES_GEN) - - coccicheck: coccicheck-test --coccicheck: $(COCCICHECK) -+coccicheck: $(COCCICHECK_OK) - - # See contrib/coccinelle/README - coccicheck-pending: coccicheck-test --coccicheck-pending: $(COCCICHECK_PENDING) -+coccicheck-pending: $(COCCICHECK_PENDING_OK) - - .PHONY: coccicheck coccicheck-pending - @@ Makefile: profile-clean: cocciclean: @@ Makefile: profile-clean: clean: profile-clean coverage-clean cocciclean $(RM) -r .build - ## contrib/coccinelle/the_repository.pending.cocci ## -@@ contrib/coccinelle/the_repository.pending.cocci: expression E; - - @@ - expression E; --expression F; - @@ - - has_object_file_with_flags( - + repo_has_object_file_with_flags(the_repository, - ## shared.mak ## @@ shared.mak: ifndef V QUIET_SP = @echo ' ' SP $<; @@ shared.mak: ifndef V + QUIET_SPATCH = @echo ' ' SPATCH $@; QUIET_SPATCH_T = @echo ' ' SPATCH TEST $(@:.build/%=%); + QUIET_SPATCH_M = @echo ' ' SPATCH MERGE $$@; -+ QUIET_SPATCH_R = @echo ' ' SPATCH RESULT $<; ## Used in "Documentation/Makefile" QUIET_ASCIIDOC = @echo ' ' ASCIIDOC $@; 4: cda20790146 ! 8: 536dce45eef cocci: make incremental compilation even faster @@ Metadata Author: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> ## Commit message ## - cocci: make incremental compilation even faster + cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES - Make the incremental compilation of "make coccicheck" even faster, at - the cost of losing dependency discovery for a small whitelist of *.c - files which we can't create *.o files for. + Improve the incremental rebuilding support of "coccicheck" by + piggy-backing on the computed dependency information of the + corresponding *.o file, rather than rebuilding all <RULE>/<FILE> pairs + if either their corresponding file changes, or if any header changes. - This will break obscure edge cases since e.g.: + This in effect uses the same method that the "sparse" target was made + to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, + 2021-09-23), except that the dependency on the *.o file isn't a hard + one, we check with $(wildcard) if the *.o file exists, and if so we'll + depend on it. - time make -W block-sha1/sha1.h coccicheck + This means that the common case of: - Would correctly re-build before, as block-sha1/sha1.c uses - block-sha1/sha1.h, but as it and these other - $(USE_TINY_FOUND_H_SOURCES) files depended on the union of all such - headers (and more, all of $(FOUND_H_SOURCES)) the above command would - take ~10 seconds before. + make + make coccicheck - As noted in the preceding commit these obscure cases were broken - before the preceding commit fixed them. I think it makes sense to make - a correctness v.s. performance trade-off here. By doing this it - becomes viable to e.g. run "make coccicheck" in the same loop that - merges various topic branches down (along with "ccache" sped-up - compilation). + Will benefit from incremental rebuilding, now changing e.g. a header + will only re-run "spatch" on those those *.c files that make use of + it: + + By depending on the *.o we piggy-back on + COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the + "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that + for the *.sp files. E.g.: + + make contrib/coccinelle/free.cocci.patch + make -W column.h contrib/coccinelle/free.cocci.patch + + Will take around 15 seconds for the second command on my 8 core box if + I didn't run "make" beforehand to create the *.o files. But around 2 + seconds if I did and we have those "*.o" files. + + Notes about the approach of piggy-backing on *.o for dependencies: + + * It *is* a trade-off since we'll pay the extra cost of running the C + compiler, but we're probably doing that anyway. The compiler is much + faster than "spatch", so even though we need to re-compile the *.o to + create the dependency info for the *.c for "spatch" it's + faster (especially if using "ccache"). + + * There *are* use-cases where some would like to have *.o files + around, but to have the "make coccicheck" ignore them. See: + https://lore.kernel.org/git/20220826104312.GJ1735@xxxxxxxxxx/ + + For those users a: + + make + make coccicheck SPATCH_USE_O_DEPENDENCIES= + + Will avoid considering the *.o files. + + * If that *.o file doesn't exist we'll depend on an intermediate file + of ours which in turn depends on $(FOUND_H_SOURCES). + + This covers both an initial build, or where "coccicheck" is run + without running "all" beforehand, and because we run "coccicheck" + on e.g. files in compat/* that we don't know how to build unless + the requisite flag was provided to the Makefile. + + Most of the runtime of "incremental" runs is now spent on various + compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and + therefore conflate whether we *can* compile an object and generate + dependency information for it with whether we'd like to link it + into our binary. + + Before this change the distinction didn't matter, but now one way + to make this even faster on incremental builds would be to peel + those concerns apart so that we can see that e.g. compat/mmap.c + doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> ## Makefile ## -@@ Makefile: COCCICHECK_OK = $(COCCICHECK:%=.build/%.ok) - COCCICHECK_PENDING_OK = $(COCCICHECK_PENDING:%=.build/%.ok) - COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) - -+USE_TINY_FOUND_H_SOURCES = -+USE_TINY_FOUND_H_SOURCES += compat/% -+USE_TINY_FOUND_H_SOURCES += block-sha1/% -+USE_TINY_FOUND_H_SOURCES += ppc/% -+ -+TINY_FOUND_H_SOURCES = -+TINY_FOUND_H_SOURCES += cache.h -+TINY_FOUND_H_SOURCES += git-compat-util.h -+TINY_FOUND_H_SOURCES += strbuf.h -+ - .build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) +@@ Makefile: COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ -@@ Makefile: define cocci-rule - # 2 = $(2) + ++SPATCH_USE_O_DEPENDENCIES = yes ++ifeq ($(COMPUTE_HEADER_DEPENDENCIES),no) ++SPATCH_USE_O_DEPENDENCIES = ++endif ++ + define cocci-rule + + ## Rule for .build/$(1).patch/$(2); Params: + # $(1) = e.g. "free.cocci" + # $(2) = e.g. "grep.c" ++# $(3) = e.g. "grep.o" COCCI_$(1:contrib/coccinelle/%.cocci=%) += .build/$(1).patch/$(2) .build/$(1).patch/$(2): GIT-SPATCH-DEFINES --.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) -+.build/$(1).patch/$(2): $(if $(wildcard $(3)),$(3),$(if $(filter $(USE_TINY_FOUND_H_SOURCES),$(3)),$(TINY_FOUND_H_SOURCES),.build/contrib/coccinelle/FOUND_H_SOURCES)) +-.build/$(1).patch/$(2): .build/contrib/coccinelle/FOUND_H_SOURCES ++.build/$(1).patch/$(2): $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) .build/$(1).patch/$(2): $(1) .build/$(1).patch/$(2): .build/$(1).patch/% : % $$(call mkdir_p_parent_template) +@@ Makefile: endef + + define cocci-matrix-2 + +-$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s))) ++$(foreach s,$(COCCI_SOURCES),$(call cocci-rule,$(c),$(s),$(s:%.c=%.o))) + endef + define cocci-matrix-1 + $(foreach c,$(COCCI),$(call cocci-matrix-2,$(c))) 5: ce4734e5d79 ! 9: 2b978676a56 spatchcache: add a ccache-alike for "spatch" @@ Commit message Add a rather trivial "spatchcache", with this running e.g.: - make clean - make contrib/coccinelle/free.cocci.patch + make cocciclean + make contrib/coccinelle/free.cocci.patch \ + SPATCH=contrib/coccicheck/spatchcache \ + SPATCH_FLAGS=--very-quiet - Is cut down from ~20s to ~3s on my system (and much of it is fixable - shell overhead). This uses "redis" as a cache by default, but it's - configurable. See the embedded documentation. + Is cut down from ~20s to ~5s on my system. Much of that is either + fixable shell overhead, or the around 40 files we "CANTCACHE" (see the + implementation). + + This uses "redis" as a cache by default, but it's configurable. See + the embedded documentation. This is *not* like ccache in that we won't cache failed spatch invocations, or those where spatch suggests changes for us. Those cases are so rare that I didn't think it was worth the bother, by far - the most common case is that it has no suggested changes. + the most common case is that it has no suggested changes. We'll also + refuse to cache any "spatch" invocation that has output on stderr, + which means that "--very-quiet" must be added to "SPATCH_FLAGS". Because we narrow the cache to that we don't need to save away stdout, stderr & the exit code. We simply cache the cases where we had no @@ contrib/coccinelle/spatchcache (new) +# driving "spatch" in git.git, in particular if we invoke: +# +# make -+# make coccicheck ++# make coccicheck SPATCH_FLAGS=--very-quiet +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we @@ contrib/coccinelle/spatchcache (new) +# "contrib/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# ++# Note that the "--very-quiet" flag is currently critical. The cache ++# will refuse to cache anything that has output on STDERR (which might ++# be errors from spatch). The STDERR (and exit code) could in ++# principle be cached (as with ccache), but then the simple structure ++# in the Redis cache would need to change, so just supply ++# "--very-quiet" for now. ++# +# To use this, simply set SPATCH to +# contrib/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] -+# # path to a custom spatch ++# # Optional: path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): @@ contrib/coccinelle/spatchcache (new) +# debug = false +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT -+# or MISS. ++# or MISS. To debug whether the cache is working do that, and run e.g.: ++# ++# redis-cli FLUSHALL ++# <make && make coccicheck, as above> ++# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c ++# 600 CANTCACHE ++# 7365 MISS ++# 7365 SET ++# ++# A subsequent "make cocciclean && make coccicheck" should then have ++# all "HIT"'s and "CANTCACHE"'s. + +set -e + @@ contrib/coccinelle/spatchcache (new) + +if ! test -f "$dep" +then -+ trace_it "$0: have no '$dep' for '$arg_file', can't cache!" ++ trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" + exec "$spatch" "$@" +fi + @@ contrib/coccinelle/spatchcache (new) + exit 128 + fi +else -+ trace_it "$0: NO CACHE (have changes): for '$arg_file' with '$arg_sp'" ++ trace_it "$0: NOCACHE (have changes): for '$arg_file' with '$arg_sp'" + cat "$out" + cat "$err" >&2 + rm -f "$out" "$err" -- 2.37.3.1420.g76f8a3d556c
