This topic is split-up from v4 of ab/make-dependency[1], and is the last topic I've split up from that which I'll submit before the outstanding ones land. This speeds up noop runs of "make" by a lot. After a "make" running a "make -j1" with this is ~1.5 faster than on "master"[2], and around 3x as fast with "make -j1 NO_TCLTK=Y" (the TCL part takes a lot of time, but that's another matter). As shown in 8/8 the only meaningful change to existing Makefile rules makes things simpler, while making them much faster. In 3/8 a rather trivial change to use ":=" instead of "=" for a "$(shell)" assignment likewise speeds things up. The "shared.mak" introduced while getting to 8/8 will then be used by follow-on topics to make introducing shared behavior (particularly via $(call), which we can now rely on) easier. The range-diff to [1] shows changes from that topic. Most of them are from pulling this topic out of those larger changes. I also re-ran all the benchmarks & updated those in the commit messages. 1. https://lore.kernel.org/git/cover-v4-00.23-00000000000-20211117T101807Z-avarab@xxxxxxxxx/ 2. $ git -c hyperfine.hook.setup= hyperfine -L rev origin/master,HEAD~0 -s 'make' 'make -j1' --warmup 1 Benchmark 1: make -j1' in 'origin/master Time (mean ± σ): 563.1 ms ± 14.5 ms [User: 453.3 ms, System: 117.6 ms] Range (min … max): 542.7 ms … 595.0 ms 10 runs Benchmark 2: make -j1' in 'HEAD~0 Time (mean ± σ): 362.5 ms ± 4.5 ms [User: 299.7 ms, System: 73.9 ms] Range (min … max): 356.9 ms … 369.4 ms 10 runs Summary 'make -j1' in 'HEAD~0' ran 1.55 ± 0.04 times faster than 'make -j1' in 'origin/master' 3. git -c hyperfine.hook.setup= hyperfine -L rev origin/master,HEAD~0 -s 'make' 'make -j1 NO_TCLTK=Y' --warmup 1 Benchmark 1: make -j1 NO_TCLTK=Y' in 'origin/master Time (mean ± σ): 291.1 ms ± 24.1 ms [User: 225.6 ms, System: 68.9 ms] Range (min … max): 275.0 ms … 353.9 ms 10 runs Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0 Time (mean ± σ): 95.8 ms ± 0.7 ms [User: 81.9 ms, System: 18.1 ms] Range (min … max): 94.8 ms … 98.4 ms 31 runs Summary 'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran 3.04 ± 0.25 times faster than 'make -j1 NO_TCLTK=Y' in 'origin/master' Ævar Arnfjörð Bjarmason (8): Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it Makefile: disable GNU make built-in wildcard rules Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) Makefile: move ".SUFFIXES" rule to shared.mak Makefile: move $(comma), $(empty) and $(space) to shared.mak Makefile: add "$(QUIET)" boilerplate to shared.mak Makefile: use $(wspfx) for $(QUIET...) in shared.mak Makefiles: add and use wildcard "mkdir -p" template Documentation/Makefile | 63 +++------------------- Makefile | 118 +++++++++++++---------------------------- config.mak.uname | 1 - shared.mak | 109 +++++++++++++++++++++++++++++++++++++ t/Makefile | 3 ++ t/interop/Makefile | 3 ++ templates/Makefile | 8 ++- 7 files changed, 160 insertions(+), 145 deletions(-) create mode 100644 shared.mak Range-diff: 1: 1621ca72c1d < -: ----------- Makefile: don't invoke msgfmt with --statistics 2: b7c36c9fea0 < -: ----------- Makefile: don't set up "perl/build" rules under NO_PERL=Y 3: 29b000eb0f1 < -: ----------- Makefile: use "=" not ":=" for po/* and perl/* 4: daead5ec293 < -: ----------- Makefile: clean perl/build/ even with NO_PERL=Y 5: 3c987590740 < -: ----------- Makefile: remove "mv $@ $@+" dance redundant to .DELETE_ON_ERROR 6: b57f582ccd3 < -: ----------- Makefile: guard Perl-only variable assignments 7: fcdee92f64c < -: ----------- Makefile: change "ifndef NO_PERL" to "ifdef NO_PERL" 8: 1e25b532ca2 < -: ----------- Makefile: adjust Perl-related comments & whitespace 9: 77d9855bfcf < -: ----------- Makefile: correct "GIT-PERL-{DEFINES,HEADER}" dependency graph 10: 6004cdcd8d9 < -: ----------- Makefile: create a GIT-PYTHON-DEFINES, like "PERL" 11: 17b30e96057 < -: ----------- Makefile: stop needing @@GIT_VERSION@@ in *.perl scripts 12: 30ddf7da2c8 ! 1: f74b47662b7 Makefiles: add "shared.mak", move ".DELETE_ON_ERROR" to it @@ shared.mak (new) +# +# info make --index-search=.DELETE_ON_ERROR +.DELETE_ON_ERROR: + + ## t/Makefile ## +@@ ++# Import tree-wide shared Makefile behavior and libraries ++include ../shared.mak ++ + # Run tests + # + # Copyright (c) 2005 Junio C Hamano + + ## t/interop/Makefile ## +@@ ++# Import tree-wide shared Makefile behavior and libraries ++include ../../shared.mak ++ + -include ../../config.mak + export GIT_TEST_OPTIONS + + + ## templates/Makefile ## +@@ ++# Import tree-wide shared Makefile behavior and libraries ++include ../shared.mak ++ + # make and install sample templates + + ifndef V 21: dd569a59c74 ! 2: b0c63abe091 Makefile: disable GNU make built-in wildcard rules @@ Commit message Makefile: disable GNU make built-in wildcard rules Override built-in rules of GNU make that use a wildcard target. This - speeds things up significantly as we don't need to stat() so many - files just in case we'd be able to retrieve their contents from RCS or - SCCS. See [1] for an old mailing list discussion about how to disable - these. - - This gives us a noticeable speedup on a no-op run: - - $ git hyperfine -L rev HEAD~1,HEAD~0 -s 'make -j8 all NO_TCLTK=Y' 'make NO_TCLTK=Y' --warmup 10 -M 10 - Benchmark 1: make NO_TCLTK=Y' in 'HEAD~1 - Time (mean ± σ): 182.2 ms ± 4.1 ms [User: 146.8 ms, System: 49.5 ms] - Range (min … max): 179.6 ms … 193.4 ms 10 runs - - Benchmark 2: make NO_TCLTK=Y' in 'HEAD~0 - Time (mean ± σ): 167.9 ms ± 1.3 ms [User: 127.8 ms, System: 55.6 ms] - Range (min … max): 166.1 ms … 170.4 ms 10 runs - - Summary - 'make NO_TCLTK=Y' in 'HEAD~0' ran - 1.09 ± 0.03 times faster than 'make NO_TCLTK=Y' in 'HEAD~1' - - Running the same except with 'strace -c -S calls make' as the - benchmark command shows (under --show-output) that we went from ~7716 - syscalls to ~7519, mostly a reduction in [l]stat(). - - We could also invoke make with "-r" by setting "MAKEFLAGS = -r" early, - adding a "-r" variant to the above benchmark shows that it may be 1.01 - or so faster (but in my tests, always with a much bigger error - bar). But doing so is a much bigger hammer, since it will disable all - built-in rules, some (all?) of which can be seen with: + can speeds things up significantly as we don't need to stat() so many + files. GNU make does that by default to see if it can retrieve their + contents from RCS or SCCS. See [1] for an old mailing list discussion + about how to disable these. + + The speed-up may wary. I've seen 1-10% depending on the speed of the + local disk, caches, -jN etc. Running: + + strace -f -c -S calls make -j1 NO_TCLTK=Y + + Shows that we reduce the number of syscalls we make, mostly in "stat" + calls. + + We could also invoke make with "-r" by setting "MAKEFLAGS = -r" + early. Doing so might make us a bit faster still. But doing so is a + much bigger hammer, since it will disable all built-in rules, + some (all?) of which can be seen with: make -f/dev/null -p | grep -v -e ^# -e ^$ 22: 4168a7e3b30 ! 3: c6c6f7cf8d8 Makefile: define $(LIB_H) in terms of $(FIND_SOURCE_FILES) @@ Commit message This speeds things up a lot: - $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -j8 all NO_TCLTK=Y' 'make NO_TCLTK=Y' --war - mup 10 -M 10 - Benchmark 1: make NO_TCLTK=Y' in 'HEAD~1 - Time (mean ± σ): 99.5 ms ± 0.5 ms [User: 83.4 ms, System: 20.7 ms] - Range (min … max): 98.8 ms … 100.2 ms 10 runs + $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make NO_TCLTK=Y' 'make -j1 NO_TCLTK=Y' --warmup 10 -M 10 + Benchmark 1: make -j1 NO_TCLTK=Y' in 'HEAD~1 + Time (mean ± σ): 159.9 ms ± 6.8 ms [User: 137.2 ms, System: 28.0 ms] + Range (min … max): 154.6 ms … 175.9 ms 10 runs - Benchmark 2: make NO_TCLTK=Y' in 'HEAD~0 - Time (mean ± σ): 69.4 ms ± 0.5 ms [User: 58.8 ms, System: 14.2 ms] - Range (min … max): 68.9 ms … 70.3 ms 10 runs + Benchmark 2: make -j1 NO_TCLTK=Y' in 'HEAD~0 + Time (mean ± σ): 100.0 ms ± 1.3 ms [User: 84.2 ms, System: 20.2 ms] + Range (min … max): 98.8 ms … 102.8 ms 10 runs Summary - 'make NO_TCLTK=Y' in 'HEAD~0' ran - 1.43 ± 0.01 times faster than 'make NO_TCLTK=Y' in 'HEAD~1' + 'make -j1 NO_TCLTK=Y' in 'HEAD~0' ran + 1.60 ± 0.07 times faster than 'make -j1 NO_TCLTK=Y' in 'HEAD~1' Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> @@ Makefile: GENERATED_H += hook-list.h LIB_OBJS += abspath.o LIB_OBJS += add-interactive.o @@ Makefile: perl/build/man/man3/Git.3pm: perl/Git.pm - $(call mkdir_p_parent_template) - $(QUIET_GEN)pod2man $< $@ + $(QUIET_GEN)mkdir -p $(dir $@) && \ + pod2man $< $@ -FIND_SOURCE_FILES = ( \ - git ls-files \ 23: 48a3927d972 ! 4: ed64cd1bd4a Makefile: move ".SUFFIXES" rule to shared.mak @@ Commit message This doesn't benefit the main Makefile at all, since it already had the rule, but since we're including shared.mak in other Makefiles - starts to benefit them. E.g. running the 'man" target is now ~1.3x - faster: + starts to benefit them. E.g. running the 'man" target is now faster: - $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -j8 -C Documentation man' 'make -C Documentation man' --warmup 10 -M 10 - Benchmark 1: make -C Documentation man' in 'HEAD~1 - Time (mean ± σ): 87.2 ms ± 1.0 ms [User: 79.3 ms, System: 10.8 ms] - Range (min … max): 86.3 ms … 89.7 ms 10 runs + $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation man' 'make -C Documentation -j1 man' + Benchmark 1: make -C Documentation -j1 man' in 'HEAD~1 + Time (mean ± σ): 121.7 ms ± 8.8 ms [User: 105.8 ms, System: 18.6 ms] + Range (min … max): 112.8 ms … 148.4 ms 26 runs - Benchmark 2: make -C Documentation man' in 'HEAD~0 - Time (mean ± σ): 64.5 ms ± 0.6 ms [User: 54.5 ms, System: 13.0 ms] - Range (min … max): 63.8 ms … 65.7 ms 10 runs + Benchmark 2: make -C Documentation -j1 man' in 'HEAD~0 + Time (mean ± σ): 97.5 ms ± 8.0 ms [User: 80.1 ms, System: 20.1 ms] + Range (min … max): 89.8 ms … 111.8 ms 32 runs Summary - 'make -C Documentation man' in 'HEAD~0' ran - 1.35 ± 0.02 times faster than 'make -C Documentation man' in 'HEAD~1' + 'make -C Documentation -j1 man' in 'HEAD~0' ran + 1.25 ± 0.14 times faster than 'make -C Documentation -j1 man' in 'HEAD~1' Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> 13: f378a7dc35e = 5: 1749085b929 Makefile: move $(comma), $(empty) and $(space) to shared.mak 14: 13cbb851d32 < -: ----------- Makefile: re-add and use the "shellquote" macros 15: 337953e4994 < -: ----------- Makefile: add a "TRACK_template" for GIT-*{FLAGS,DEFINES,...} 16: 5bb597c1993 ! 6: c25284b24cf Makefile: add "$(QUIET)" boilerplate to shared.mak @@ config.mak.uname: vcxproj: ## shared.mak ## -@@ shared.mak: space = $(empty) $(empty) - wspfx = $(space)$(space)$(space) - wspfx_sq = $(call shellquote,$(wspfx)) - +@@ + comma = , + empty = + space = $(empty) $(empty) ++ +### Quieting +## common +QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir @@ shared.mak: space = $(empty) $(empty) + export V +endif +endif -+ - ### Templates - - ## Template for making a GIT-SOMETHING, which changes if a ## templates/Makefile ## @@ 17: 3c4d0589667 < -: ----------- Makefile: use $(wspfx) for $(QUIET...) in shared.mak -: ----------- > 7: 3daef7672be Makefile: use $(wspfx) for $(QUIET...) in shared.mak 18: be5882b2c99 ! 8: aca560ca410 Makefiles: add and use wildcard "mkdir -p" template @@ Commit message shows that this is faster, in addition to being less verbose and more reliable (this uses my "git-hyperfine" thin wrapper for "hyperfine"[3]): - $ git hyperfine -L rev HEAD~0,HEAD~1 -b 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation lint-docs' - Benchmark 1: make -C Documentation lint-docs' in 'HEAD~0 - Time (mean ± σ): 2.129 s ± 0.011 s [User: 1.840 s, System: 0.321 s] - Range (min … max): 2.121 s … 2.158 s 10 runs + $ git -c hyperfine.hook.setup= hyperfine -L rev HEAD~1,HEAD~0 -s 'make -C Documentation lint-docs' -p 'rm -rf Documentation/.build' 'make -C Documentation -j1 lint-docs' + Benchmark 1: make -C Documentation -j1 lint-docs' in 'HEAD~1 + Time (mean ± σ): 2.914 s ± 0.062 s [User: 2.449 s, System: 0.489 s] + Range (min … max): 2.834 s … 3.020 s 10 runs - Benchmark 2: make -C Documentation lint-docs' in 'HEAD~1 - Time (mean ± σ): 2.659 s ± 0.002 s [User: 2.306 s, System: 0.397 s] - Range (min … max): 2.657 s … 2.662 s 10 runs + Benchmark 2: make -C Documentation -j1 lint-docs' in 'HEAD~0 + Time (mean ± σ): 2.315 s ± 0.062 s [User: 1.950 s, System: 0.386 s] + Range (min … max): 2.229 s … 2.397 s 10 runs Summary - 'make -C Documentation lint-docs' in 'HEAD~0' ran - 1.25 ± 0.01 times faster than 'make -C Documentation lint-docs' in 'HEAD~1' + 'make -C Documentation -j1 lint-docs' in 'HEAD~0' ran + 1.26 ± 0.04 times faster than 'make -C Documentation -j1 lint-docs' in 'HEAD~1' So let's use that pattern both for the "lint-docs" target, and a few miscellaneous other targets. @@ Makefile: all:: $(MOFILES) + $(call mkdir_p_parent_template) + $(QUIET_MSGFMT)$(MSGFMT) -o $@ $< - ifndef NO_PERL - LIB_PERL = $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) -@@ Makefile: LIB_CPAN = $(wildcard perl/FromCPAN/*.pm perl/FromCPAN/*/*.pm) - LIB_CPAN_GEN = $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_CPAN)) + LIB_PERL := $(wildcard perl/Git.pm perl/Git/*.pm perl/Git/*/*.pm perl/Git/*/*/*.pm) + LIB_PERL_GEN := $(patsubst perl/%.pm,perl/build/lib/%.pm,$(LIB_PERL)) +@@ Makefile: NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) + endif perl/build/lib/%.pm: perl/%.pm GIT-PERL-DEFINES - $(QUIET_GEN)mkdir -p $(dir $@) && \ @@ Makefile: LIB_CPAN = $(wildcard perl/FromCPAN/*.pm perl/FromCPAN/*/*.pm) + $(QUIET_GEN) \ sed -e 's|@@LOCALEDIR@@|$(perl_localedir_SQ)|g' \ -e 's|@@NO_GETTEXT@@|$(NO_GETTEXT_SQ)|g' \ - -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(call shq,$(NO_PERL_CPAN_FALLBACKS))|g' \ -@@ Makefile: endif + -e 's|@@NO_PERL_CPAN_FALLBACKS@@|$(NO_PERL_CPAN_FALLBACKS_SQ)|g' \ + < $< > $@ - # install-man depends on Git.3pm even with NO_PERL=Y perl/build/man/man3/Git.3pm: perl/Git.pm - $(QUIET_GEN)mkdir -p $(dir $@) && \ - pod2man $< $@ + $(call mkdir_p_parent_template) + $(QUIET_GEN)pod2man $< $@ - FIND_SOURCE_FILES = ( \ - git ls-files \ + $(ETAGS_TARGET): $(FOUND_SOURCE_FILES) + $(QUIET_GEN)$(RM) $@+ && \ @@ Makefile: test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_NEED_X) $( all:: $(TEST_PROGRAMS) $(test_bindir_programs) @@ Makefile: test_bindir_programs := $(patsubst %,bin-wrappers/%,$(BINDIR_PROGRAMS_ ## shared.mak ## @@ shared.mak: ifndef V QUIET = @ - QUIET_GEN = @echo $(wspfx_sq) GEN $@; + QUIET_GEN = @echo $(wspfx_SQ) GEN $@; -+ QUIET_MKDIR_P_PARENT = @echo $(wspfx_sq) MKDIR -p $(@D); ++ QUIET_MKDIR_P_PARENT = @echo $(wspfx_SQ) MKDIR -p $(@D); + ## Used in "Makefile" - QUIET_CC = @echo $(wspfx_sq) CC $@; - QUIET_AR = @echo $(wspfx_sq) AR $@; + QUIET_CC = @echo $(wspfx_SQ) CC $@; + QUIET_AR = @echo $(wspfx_SQ) AR $@; @@ shared.mak: ifndef V + export V endif endif - -+## Helpers ++ ++### Templates ++ ++## mkdir_p_parent: lazily "mkdir -p" the path needed for a $@ ++## file. Uses $(wildcard) to avoid the "mkdir -p" if it's not ++## needed. ++## ++## Is racy, but in a good way; we might redundantly (and safely) ++## "mkdir -p" when running in parallel, but won't need to exhaustively ++## individual rules for "a" -> "prefix" -> "dir" -> "file" if given a ++## "a/prefix/dir/file". This can instead be inserted at the start of ++## the "a/prefix/dir/file" rule. +define mkdir_p_parent_template +$(if $(wildcard $(@D)),,$(QUIET_MKDIR_P_PARENT)$(shell mkdir -p $(@D))) +endef -+ - ### Templates - - ## Template for making a GIT-SOMETHING, which changes if a 19: 2710f8af6cd < -: ----------- Makefile: correct the dependency graph of hook-list.h 20: 59f22a0269a < -: ----------- Makefile: use $(file) I/O instead of "FORCE" when possible -- 2.34.1.1119.g7a3fc8778ee
