[PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is a second attempt at redesigning Git's repository maintenance
patterns. The first attempt [1] included a way to run jobs in the background
using a long-lived process; that idea was rejected and is not included in
this series. A future series will use the OS to handle scheduling tasks.

[1] 
https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@xxxxxxxxx/

As mentioned before, git gc already plays the role of maintaining Git
repositories. It has accumulated several smaller pieces in its long history,
including:

 1. Repacking all reachable objects into one pack-file (and deleting
    unreachable objects).
 2. Packing refs.
 3. Expiring reflogs.
 4. Clearing rerere logs.
 5. Updating the commit-graph file.
 6. Pruning worktrees.

While expiring reflogs, clearing rererelogs, and deleting unreachable
objects are suitable under the guise of "garbage collection", packing refs
and updating the commit-graph file are not as obviously fitting. Further,
these operations are "all or nothing" in that they rewrite almost all
repository data, which does not perform well at extremely large scales.
These operations can also be disruptive to foreground Git commands when git
gc --auto triggers during routine use.

This series does not intend to change what git gc does, but instead create
new choices for automatic maintenance activities, of which git gc remains
the only one enabled by default.

The new maintenance tasks are:

 * 'commit-graph' : write and verify a single layer of an incremental
   commit-graph.
 * 'loose-objects' : prune packed loose objects, then create a new pack from
   a batch of loose objects.
 * 'pack-files' : expire redundant packs from the multi-pack-index, then
   repack using the multi-pack-index's incremental repack strategy.
 * 'fetch' : fetch from each remote, storing the refs in 'refs/prefetch//'.

These tasks are all disabled by default, but can be enabled with config
options or run explicitly using "git maintenance run --task=". There are
additional config options to allow customizing the conditions for which the
tasks run during the '--auto' option. ('fetch' will never run with the
'--auto' option.)

 Because 'gc' is implemented as a maintenance task, the most dramatic change
of this series is to convert the 'git gc --auto' calls into 'git maintenance
run --auto' calls at the end of some Git commands. By default, the only
change is that 'git gc --auto' will be run below an additional 'git
maintenance' process.

The 'git maintenance' builtin has a 'run' subcommand so it can be extended
later with subcommands that manage background maintenance, such as 'start',
'stop', 'pause', or 'schedule'. These are not the subject of this series, as
it is important to focus on the maintenance activities themselves.

An expert user could set up scheduled background maintenance themselves with
the current series. I have the following crontab data set up to run
maintenance on an hourly basis:

0 * * * * git -C /<path-to-repo> maintenance run --no-quiet >>/<path-to-repo>/.git/maintenance.log

My config includes all tasks except the 'gc' task. The hourly run is
over-aggressive, but is sufficient for testing. I'll replace it with daily
when I feel satisfied.

Hopefully this direction is seen as a positive one. My goal was to add more
options for expert users, along with the flexibility to create background
maintenance via the OS in a later series.

OUTLINE
=======

Patches 1-4 remove some references to the_repository in builtin/gc.c before
we start depending on code in that builtin.

Patches 5-7 create the 'git maintenance run' builtin and subcommand as a
simple shim over 'git gc' and replaces calls to 'git gc --auto' from other
commands.

Patches 8-15 create new maintenance tasks. These are the same tasks sent in
the previous RFC.

Patches 16-21 create more customization through config and perform other
polish items.

FUTURE WORK
===========

 * Add 'start', 'stop', and 'schedule' subcommands to initialize the
   commands run in the background. You can see my progress towards this goal
   here: https://github.com/gitgitgadget/git/pull/680
   
   
 * Split the 'gc' builtin into smaller maintenance tasks that are enabled by
   default, but might have different '--auto' conditions and more config
   options.
   
   
 * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'
   with use of the 'commit-graph' task.
   
   
 * Update the builtin to use struct repository *r properly, especially when
   calling subcommands.
   
   

UPDATES in V2
=============

I'm sending this between v2.28.0-rc2 adn v2.28.0 as the release things have
become a bit quiet.

 * The biggest disruption to the range-diff is that I removed the premature
   use of struct repository *r and instead continue to rely on 
   the_repository. This means several patches were dropped that did prep
   work in builtin/gc.c.
   
   
 * I dropped the task hashmap and opted for a linear scan. This task list
   will always be too small to justify the extra complication of the
   hashmap.
   
   
 * struct maintenance_opts is properly static now.
   
   
 * Some tasks are renamed: fetch -> prefetch, pack-files ->
   incremental-repack.
   
   
 * With the rename, the prefetch task uses refs/prefetch/ instead of 
   refs/hidden/.
   
   
 * A trace2 region around the task executions are added.
   
   

UPDATES in V3
=============

 * This series is now based on jk/strvec, as there are several places where
   I was adding new callers to argv_array_push* and run_command_v_opt()
   which have been replaced with strvec_push*() and run_command(), using a
   'struct child_process'.
   
   
 * I added and updates Junio's patch from jc/no-update-fetch-head into the
   proper place before the 'prefetch' task. The 'prefetch' task now uses
   --no-write-fetch-head to avoid issues with FETCH_HEAD.
   
   
 * Since there were concerns around core.multiPackIndex, I added some extra
   care checking for that config being enabled. Since that already needed to
   adjust the config value from its existing implementation in midx.c, I
   added it to repo-settings and made it enabled by default.
   
   
 * Lots of feedback from the previous round. Thanks, all! I fully expect to
   have at least one more round of feedback, but things are improving quite
   a lot.
   
   

Thanks, -Stolee

Derrick Stolee (19):
  maintenance: create basic maintenance runner
  maintenance: add --quiet option
  maintenance: replace run_auto_gc()
  maintenance: initialize task array
  maintenance: add commit-graph task
  maintenance: add --task option
  maintenance: take a lock on the objects directory
  maintenance: add prefetch task
  maintenance: add loose-objects task
  midx: enable core.multiPackIndex by default
  maintenance: add incremental-repack task
  maintenance: auto-size incremental-repack batch
  maintenance: create maintenance.<task>.enabled config
  maintenance: use pointers to check --auto
  maintenance: add auto condition for commit-graph task
  maintenance: create auto condition for loose-objects
  maintenance: add incremental-repack auto condition
  midx: use start_delayed_progress()
  maintenance: add trace2 regions for task execution

Junio C Hamano (1):
  fetch: optionally allow disabling FETCH_HEAD update

 .gitignore                           |   1 +
 Documentation/config.txt             |   2 +
 Documentation/config/core.txt        |   4 +-
 Documentation/config/fetch.txt       |   7 +
 Documentation/config/maintenance.txt |  32 ++
 Documentation/fetch-options.txt      |  16 +-
 Documentation/git-clone.txt          |   6 +-
 Documentation/git-maintenance.txt    | 127 +++++
 builtin.h                            |   1 +
 builtin/am.c                         |   2 +-
 builtin/commit.c                     |   2 +-
 builtin/fetch.c                      |  25 +-
 builtin/gc.c                         | 713 +++++++++++++++++++++++++++
 builtin/merge.c                      |   2 +-
 builtin/pull.c                       |   3 +-
 builtin/rebase.c                     |   4 +-
 commit-graph.c                       |   8 +-
 commit-graph.h                       |   1 +
 git.c                                |   1 +
 midx.c                               |  23 +-
 midx.h                               |   1 +
 object.h                             |   1 +
 repo-settings.c                      |   6 +
 repository.h                         |   2 +
 run-command.c                        |  16 +-
 run-command.h                        |   2 +-
 t/t5319-multi-pack-index.sh          |  15 +-
 t/t5510-fetch.sh                     |  41 +-
 t/t5514-fetch-multiple.sh            |   2 +-
 t/t5521-pull-options.sh              |  16 +
 t/t7900-maintenance.sh               | 215 ++++++++
 31 files changed, 1240 insertions(+), 57 deletions(-)
 create mode 100644 Documentation/config/maintenance.txt
 create mode 100644 Documentation/git-maintenance.txt
 create mode 100755 t/t7900-maintenance.sh


base-commit: 90dfbf331c0ade4c15c74276c466e32e384f9ceb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-671%2Fderrickstolee%2Fmaintenance%2Fgc-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-671/derrickstolee/maintenance/gc-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/671

Range-diff vs v2:

  1:  63ec602a07 !  1:  12fe73bb72 maintenance: create basic maintenance runner
     @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
      +
      +static int maintenance_task_gc(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     ++	struct child_process child = CHILD_PROCESS_INIT;
      +
     -+	argv_array_pushl(&cmd, "gc", NULL);
     ++	child.git_cmd = 1;
     ++	strvec_push(&child.args, "gc");
      +
      +	if (opts.auto_flag)
     -+		argv_array_pushl(&cmd, "--auto", NULL);
     ++		strvec_push(&child.args, "--auto");
      +
      +	close_object_store(the_repository->objects);
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
     -+
     -+	return result;
     ++	return run_command(&child);
      +}
      +
      +static int maintenance_run(void)
     @@ t/t7900-maintenance.sh (new)
      +
      +test_description='git maintenance builtin'
      +
     -+GIT_TEST_COMMIT_GRAPH=0
     -+GIT_TEST_MULTI_PACK_INDEX=0
     -+
      +. ./test-lib.sh
      +
      +test_expect_success 'help text' '
     -+	test_must_fail git maintenance -h 2>err &&
     ++	test_expect_code 129 git maintenance -h 2>err &&
      +	test_i18ngrep "usage: git maintenance run" err
      +'
      +
     -+test_expect_success 'gc [--auto]' '
     ++test_expect_success 'run [--auto]' '
      +	GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run &&
      +	GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto &&
      +	grep ",\"gc\"]" run-no-auto.txt  &&
  2:  1d37e55cb7 !  2:  6e533e43d7 maintenance: add --quiet option
     @@ builtin/gc.c: static const char * const builtin_maintenance_usage[] = {
      @@ builtin/gc.c: static int maintenance_task_gc(void)
       
       	if (opts.auto_flag)
     - 		argv_array_pushl(&cmd, "--auto", NULL);
     + 		strvec_push(&child.args, "--auto");
      +	if (opts.quiet)
     -+		argv_array_pushl(&cmd, "--quiet", NULL);
     ++		strvec_push(&child.args, "--quiet");
       
       	close_object_store(the_repository->objects);
     - 	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     + 	return run_command(&child);
      @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)
       	static struct option builtin_maintenance_options[] = {
       		OPT_BOOL(0, "auto", &opts.auto_flag,
     @@ t/t7900-maintenance.sh: test_expect_success 'help text' '
       	test_i18ngrep "usage: git maintenance run" err
       '
       
     --test_expect_success 'gc [--auto]' '
     +-test_expect_success 'run [--auto]' '
      -	GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run &&
     -+test_expect_success 'gc [--auto|--quiet]' '
     ++test_expect_success 'run [--auto|--quiet]' '
      +	GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run --no-quiet &&
       	GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto &&
      +	GIT_TRACE2_EVENT="$(pwd)/run-quiet.txt" git maintenance run --quiet &&
  3:  f164d1a0b4 !  3:  c4674fc211 maintenance: replace run_auto_gc()
     @@ Commit message
      
          Rename run_auto_gc() to run_auto_maintenance() to be clearer what is
          happening on this call, and to expose all callers in the current diff.
     +    Rewrite the method to use a struct child_process to simplify the calls
     +    slightly.
      
          Since 'git fetch' already allows disabling the 'git gc --auto'
          subprocess, add an equivalent option with a different name to be more
     @@ Documentation/fetch-options.txt: ifndef::git-pull[]
       	Allow several <repository> and <group> arguments to be
       	specified. No <refspec>s may be specified.
       
     -+--[no-]maintenance::
     ++--[no-]auto-maintenance::
       --[no-]auto-gc::
      -	Run `git gc --auto` at the end to perform garbage collection
      -	if needed. This is enabled by default.
     -+	Run `git maintenance run --auto` at the end to perform garbage
     -+	collection if needed. This is enabled by default.
     ++	Run `git maintenance run --auto` at the end to perform automatic
     ++	repository maintenance if needed. (`--[no-]auto-gc` is a synonym.)
     ++	This is enabled by default.
       
       --[no-]write-commit-graph::
       	Write a commit-graph after fetching. This overrides the config
     @@ Documentation/git-clone.txt: repository using this option and then delete branch
      -which automatically call `git gc --auto`. (See linkgit:git-gc[1].)
      -If these objects are removed and were referenced by the cloned repository,
      -then the cloned repository will become corrupt.
     -+which automatically call `git maintenance run --auto` and `git gc --auto`.
     -+(See linkgit:git-maintenance[1] and linkgit:git-gc[1].) If these objects
     -+are removed and were referenced by the cloned repository, then the cloned
     -+repository will become corrupt.
     ++which automatically call `git maintenance run --auto`. (See
     ++linkgit:git-maintenance[1].) If these objects are removed and were referenced
     ++by the cloned repository, then the cloned repository will become corrupt.
       +
       Note that running `git repack` without the `--local` option in a repository
       cloned with `--shared` will copy objects from the source repository into a pack
     @@ builtin/fetch.c: static struct option builtin_fetch_options[] = {
       	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
       			N_("report that we have only objects reachable from this object")),
       	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
     -+	OPT_BOOL(0, "maintenance", &enable_auto_gc,
     ++	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
      +		 N_("run 'maintenance --auto' after fetching")),
       	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
      -		 N_("run 'gc --auto' after fetching")),
     @@ run-command.c: int run_processes_parallel_tr2(int n, get_next_task_fn get_next_t
      -int run_auto_gc(int quiet)
      +int run_auto_maintenance(int quiet)
       {
     - 	struct argv_array argv_gc_auto = ARGV_ARRAY_INIT;
     - 	int status;
     +-	struct strvec argv_gc_auto = STRVEC_INIT;
     +-	int status;
     ++	struct child_process maint = CHILD_PROCESS_INIT;
       
     --	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
     -+	argv_array_pushl(&argv_gc_auto, "maintenance", "run", "--auto", NULL);
     - 	if (quiet)
     - 		argv_array_push(&argv_gc_auto, "--quiet");
     -+	else
     -+		argv_array_push(&argv_gc_auto, "--no-quiet");
     +-	strvec_pushl(&argv_gc_auto, "gc", "--auto", NULL);
     +-	if (quiet)
     +-		strvec_push(&argv_gc_auto, "--quiet");
     +-	status = run_command_v_opt(argv_gc_auto.items, RUN_GIT_CMD);
     +-	strvec_clear(&argv_gc_auto);
     +-	return status;
     ++	maint.git_cmd = 1;
     ++	strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL);
     ++	strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet");
      +
     - 	status = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);
     - 	argv_array_clear(&argv_gc_auto);
     - 	return status;
     ++	return run_command(&maint);
     + }
      
       ## run-command.h ##
      @@ run-command.h: int run_hook_ve(const char *const *env, const char *name, va_list args);
  4:  8e260bccf1 !  4:  b9332c1318 maintenance: initialize task array
     @@ Commit message
          future command-line argument) along with a function pointer to its
          implementation and a boolean for whether the step is enabled.
      
     -    A list of pointers to these structs are initialized with the full list
     -    of implemented tasks along with a default order. For now, this list only
     -    contains the "gc" task. This task is also the only task enabled by
     -    default.
     +    A list these structs are initialized with the full list of implemented
     +    tasks along with a default order. For now, this list only contains the
     +    "gc" task. This task is also the only task enabled by default.
      
     +    The run subcommand will return a nonzero exit code if any task fails.
     +    However, it will attempt all tasks in its loop before returning with the
     +    failure. Also each failed task will send an error message.
     +
     +    Helped-by: Taylor Blau <me@xxxxxxxxxxxx>
     +    Helped-by: Junio C Hamano <gitster@xxxxxxxxx>
          Signed-off-by: Derrick Stolee <dstolee@xxxxxxxxxxxxx>
      
       ## builtin/gc.c ##
     -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
     - 	return 0;
     - }
     - 
     -+#define MAX_NUM_TASKS 1
     -+
     - static const char * const builtin_maintenance_usage[] = {
     - 	N_("git maintenance run [<options>]"),
     - 	NULL
      @@ builtin/gc.c: static int maintenance_task_gc(void)
     - 	return result;
     + 	return run_command(&child);
       }
       
      +typedef int maintenance_task_fn(void);
     @@ builtin/gc.c: static int maintenance_task_gc(void)
      +	unsigned enabled:1;
      +};
      +
     -+static struct maintenance_task *tasks[MAX_NUM_TASKS];
     -+static int num_tasks;
     ++enum maintenance_task_label {
     ++	TASK_GC,
     ++
     ++	/* Leave as final value */
     ++	TASK__COUNT
     ++};
     ++
     ++static struct maintenance_task tasks[] = {
     ++	[TASK_GC] = {
     ++		"gc",
     ++		maintenance_task_gc,
     ++		1,
     ++	},
     ++};
      +
       static int maintenance_run(void)
       {
     @@ builtin/gc.c: static int maintenance_task_gc(void)
      +	int i;
      +	int result = 0;
      +
     -+	for (i = 0; !result && i < num_tasks; i++) {
     -+		if (!tasks[i]->enabled)
     ++	for (i = 0; i < TASK__COUNT; i++) {
     ++		if (!tasks[i].enabled)
      +			continue;
     -+		result = tasks[i]->fn();
     ++
     ++		if (tasks[i].fn()) {
     ++			error(_("task '%s' failed"), tasks[i].name);
     ++			result = 1;
     ++		}
      +	}
      +
      +	return result;
     -+}
     -+
     -+static void initialize_tasks(void)
     -+{
     -+	int i;
     -+	num_tasks = 0;
     -+
     -+	for (i = 0; i < MAX_NUM_TASKS; i++)
     -+		tasks[i] = xcalloc(1, sizeof(struct maintenance_task));
     -+
     -+	tasks[num_tasks]->name = "gc";
     -+	tasks[num_tasks]->fn = maintenance_task_gc;
     -+	tasks[num_tasks]->enabled = 1;
     -+	num_tasks++;
       }
       
       int cmd_maintenance(int argc, const char **argv, const char *prefix)
     -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)
     - 				   builtin_maintenance_options);
     - 
     - 	opts.quiet = !isatty(2);
     -+	initialize_tasks();
     - 
     - 	argc = parse_options(argc, argv, prefix,
     - 			     builtin_maintenance_options,
  5:  04552b1d2e !  5:  a4d9836bed maintenance: add commit-graph task
     @@ Documentation/git-maintenance.txt: run::
       	stands for "garbage collection," but this task performs many
      
       ## builtin/gc.c ##
     -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
     - 	return 0;
     - }
     - 
     --#define MAX_NUM_TASKS 1
     -+#define MAX_NUM_TASKS 2
     - 
     - static const char * const builtin_maintenance_usage[] = {
     - 	N_("git maintenance run [<options>]"),
      @@ builtin/gc.c: static struct maintenance_opts {
       	int quiet;
       } opts;
       
      +static int run_write_commit_graph(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     ++	struct child_process child = CHILD_PROCESS_INIT;
      +
     -+	argv_array_pushl(&cmd, "commit-graph", "write",
     -+			 "--split", "--reachable", NULL);
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "commit-graph", "write",
     ++		     "--split", "--reachable", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     -+
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
     ++		strvec_push(&child.args, "--no-progress");
      +
     -+	return result;
     ++	return !!run_command(&child);
      +}
      +
      +static int run_verify_commit_graph(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     ++	struct child_process child = CHILD_PROCESS_INIT;
      +
     -+	argv_array_pushl(&cmd, "commit-graph", "verify",
     -+			 "--shallow", NULL);
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "commit-graph", "verify",
     ++		     "--shallow", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     ++		strvec_push(&child.args, "--no-progress");
      +
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
     -+
     -+	return result;
     ++	return !!run_command(&child);
      +}
      +
      +static int maintenance_task_commit_graph(void)
     @@ builtin/gc.c: static struct maintenance_opts {
      +	struct repository *r = the_repository;
      +	char *chain_path;
      +
     -+	/* Skip commit-graph when --auto is specified. */
     -+	if (opts.auto_flag)
     -+		return 0;
     -+
      +	close_object_store(r->objects);
      +	if (run_write_commit_graph()) {
      +		error(_("failed to write commit-graph"));
     @@ builtin/gc.c: static struct maintenance_opts {
      +
       static int maintenance_task_gc(void)
       {
     - 	int result;
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	tasks[num_tasks]->fn = maintenance_task_gc;
     - 	tasks[num_tasks]->enabled = 1;
     - 	num_tasks++;
     -+
     -+	tasks[num_tasks]->name = "commit-graph";
     -+	tasks[num_tasks]->fn = maintenance_task_commit_graph;
     -+	num_tasks++;
     - }
     + 	struct child_process child = CHILD_PROCESS_INIT;
     +@@ builtin/gc.c: struct maintenance_task {
     + 
     + enum maintenance_task_label {
     + 	TASK_GC,
     ++	TASK_COMMIT_GRAPH,
       
     - int cmd_maintenance(int argc, const char **argv, const char *prefix)
     + 	/* Leave as final value */
     + 	TASK__COUNT
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 		maintenance_task_gc,
     + 		1,
     + 	},
     ++	[TASK_COMMIT_GRAPH] = {
     ++		"commit-graph",
     ++		maintenance_task_commit_graph,
     ++	},
     + };
     + 
     + static int maintenance_run(void)
      
       ## commit-graph.c ##
      @@ commit-graph.c: static char *get_split_graph_filename(struct object_directory *odb,
     @@ commit-graph.h: struct commit;
       /*
      
       ## t/t7900-maintenance.sh ##
     -@@ t/t7900-maintenance.sh: test_expect_success 'help text' '
     - 	test_i18ngrep "usage: git maintenance run" err
     - '
     +@@ t/t7900-maintenance.sh: test_description='git maintenance builtin'
       
     --test_expect_success 'gc [--auto|--quiet]' '
     -+test_expect_success 'run [--auto|--quiet]' '
     - 	GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run --no-quiet &&
     - 	GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto &&
     - 	GIT_TRACE2_EVENT="$(pwd)/run-quiet.txt" git maintenance run --quiet &&
     + . ./test-lib.sh
     + 
     ++GIT_TEST_COMMIT_GRAPH=0
     ++
     + test_expect_success 'help text' '
     + 	test_expect_code 129 git maintenance -h 2>err &&
     + 	test_i18ngrep "usage: git maintenance run" err
  6:  a09b1c1687 !  6:  dafb0d9bbc maintenance: add --task option
     @@ Commit message
          references. We use the hashmap to match the --task=<task> arguments into
          the task struct data.
      
     +    Keep in mind that the 'enabled' member of the maintenance_task struct is
     +    a placeholder for a future 'maintenance.<task>.enabled' config option.
     +    Thus, we use the 'enabled' member to specify which tasks are run when
     +    the user does not specify any --task=<task> arguments. The 'enabled'
     +    member should be ignored if --task=<task> appears.
     +
          Signed-off-by: Derrick Stolee <dstolee@xxxxxxxxxxxxx>
      
       ## Documentation/git-maintenance.txt ##
     @@ builtin/gc.c: typedef int maintenance_task_fn(void);
       	const char *name;
       	maintenance_task_fn *fn;
      -	unsigned enabled:1;
     -+	int task_order;
      +	unsigned enabled:1,
      +		 selected:1;
     ++	int selected_order;
       };
       
     - static struct maintenance_task *tasks[MAX_NUM_TASKS];
     - static int num_tasks;
     + enum maintenance_task_label {
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 	},
     + };
       
      +static int compare_tasks_by_selection(const void *a_, const void *b_)
      +{
      +	const struct maintenance_task *a, *b;
     -+	a = (const struct maintenance_task *)a_;
     -+	b = (const struct maintenance_task *)b_;
      +
     -+	return b->task_order - a->task_order;
     ++	a = (const struct maintenance_task *)&a_;
     ++	b = (const struct maintenance_task *)&b_;
     ++
     ++	return b->selected_order - a->selected_order;
      +}
      +
       static int maintenance_run(void)
     @@ builtin/gc.c: typedef int maintenance_task_fn(void);
       	int result = 0;
       
      +	if (opts.tasks_selected)
     -+		QSORT(tasks, num_tasks, compare_tasks_by_selection);
     ++		QSORT(tasks, TASK__COUNT, compare_tasks_by_selection);
      +
     - 	for (i = 0; !result && i < num_tasks; i++) {
     --		if (!tasks[i]->enabled)
     -+		if (opts.tasks_selected && !tasks[i]->selected)
     + 	for (i = 0; i < TASK__COUNT; i++) {
     +-		if (!tasks[i].enabled)
     ++		if (opts.tasks_selected && !tasks[i].selected)
      +			continue;
      +
     -+		if (!opts.tasks_selected && !tasks[i]->enabled)
     ++		if (!opts.tasks_selected && !tasks[i].enabled)
       			continue;
     -+
     - 		result = tasks[i]->fn();
     - 	}
       
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	num_tasks++;
     + 		if (tasks[i].fn()) {
     +@@ builtin/gc.c: static int maintenance_run(void)
     + 	return result;
       }
       
      +static int task_option_parse(const struct option *opt,
     @@ builtin/gc.c: static void initialize_tasks(void)
      +
      +	BUG_ON_OPT_NEG(unset);
      +
     -+	if (!arg || !strlen(arg)) {
     -+		error(_("--task requires a value"));
     -+		return 1;
     -+	}
     -+
      +	opts.tasks_selected++;
      +
     -+	for (i = 0; i < MAX_NUM_TASKS; i++) {
     -+		if (tasks[i] && !strcasecmp(tasks[i]->name, arg)) {
     -+			task = tasks[i];
     ++	for (i = 0; i < TASK__COUNT; i++) {
     ++		if (!strcasecmp(tasks[i].name, arg)) {
     ++			task = &tasks[i];
      +			break;
      +		}
      +	}
     @@ builtin/gc.c: static void initialize_tasks(void)
      +	}
      +
      +	task->selected = 1;
     -+	task->task_order = opts.tasks_selected;
     ++	task->selected_order = opts.tasks_selected;
      +
      +	return 0;
      +}
  7:  e9260a9c3f !  7:  1b00524da3 maintenance: take a lock on the objects directory
     @@ builtin/gc.c: static int maintenance_run(void)
      +	free(lock_path);
       
       	if (opts.tasks_selected)
     - 		QSORT(tasks, num_tasks, compare_tasks_by_selection);
     + 		QSORT(tasks, TASK__COUNT, compare_tasks_by_selection);
      @@ builtin/gc.c: static int maintenance_run(void)
     - 		result = tasks[i]->fn();
     + 		}
       	}
       
      +	rollback_lock_file(&lk);
  -:  ---------- >  8:  0e94e04dcd fetch: optionally allow disabling FETCH_HEAD update
  8:  3165b8916d !  9:  9e38ade15c maintenance: add prefetch task
     @@ Commit message
      
          2. --refmap= removes the configured refspec which usually updates
             refs/remotes/<remote>/* with the refs advertised by the remote.
     +       While this looks confusing, this was documented and tested by
     +       b40a50264ac (fetch: document and test --refmap="", 2020-01-21),
     +       including this sentence in the documentation:
     +
     +            Providing an empty `<refspec>` to the `--refmap` option
     +            causes Git to ignore the configured refspecs and rely
     +            entirely on the refspecs supplied as command-line arguments.
      
          3. By adding a new refspec "+refs/heads/*:refs/prefetch/<remote>/*"
             we can ensure that we actually load the new values somewhere in
     @@ Commit message
          4. --prune will delete the refs/prefetch/<remote> refs that no
             longer appear on the remote.
      
     +    5. --no-write-fetch-head prevents updating FETCH_HEAD.
     +
          We've been using this step as a critical background job in Scalar
          [1] (and VFS for Git). This solved a pain point that was showing up
          in user reports: fetching was a pain! Users do not like waiting to
     @@ Documentation/git-maintenance.txt: since it will not expire `.graph` files that
       the expiration delay.
       
      +prefetch::
     -+	The `fetch` task updates the object directory with the latest objects
     -+	from all registered remotes. For each remote, a `git fetch` command
     -+	is run. The refmap is custom to avoid updating local or remote
     ++	The `prefetch` task updates the object directory with the latest
     ++	objects from all registered remotes. For each remote, a `git fetch`
     ++	command is run. The refmap is custom to avoid updating local or remote
      +	branches (those in `refs/heads` or `refs/remotes`). Instead, the
      +	remote refs are stored in `refs/prefetch/<remote>/`. Also, tags are
      +	not updated.
      ++
     -+This means that foreground fetches are still required to update the
     -+remote refs, but the users is notified when the branches and tags are
     -+updated on the remote.
     ++This is done to avoid disrupting the remote-tracking branches. The end users
     ++expect these refs to stay unmoved unless they initiate a fetch.  With prefetch
     ++task, however, the objects necessary to complete a later real fetch would
     ++already be obtained, so the real fetch would go faster.  In the ideal case,
     ++it will just become an update to bunch of remote-tracking branches without
     ++any object transfer.
      +
       gc::
       	Cleanup unnecessary files and optimize the local repository. "GC"
     @@ builtin/gc.c
       
       #define FAILED_RUN "failed to run %s"
       
     -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
     - 	return 0;
     - }
     - 
     --#define MAX_NUM_TASKS 2
     -+#define MAX_NUM_TASKS 3
     - 
     - static const char * const builtin_maintenance_usage[] = {
     - 	N_("git maintenance run [<options>]"),
      @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
       	return 1;
       }
       
      +static int fetch_remote(const char *remote)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	struct strbuf refmap = STRBUF_INIT;
     ++	struct child_process child = CHILD_PROCESS_INIT;
      +
     -+	argv_array_pushl(&cmd, "fetch", remote, "--prune",
     -+			 "--no-tags", "--refmap=", NULL);
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "fetch", remote, "--prune", "--no-tags",
     ++		     "--no-write-fetch-head", "--refmap=", NULL);
      +
     -+	strbuf_addf(&refmap, "+refs/heads/*:refs/prefetch/%s/*", remote);
     -+	argv_array_push(&cmd, refmap.buf);
     ++	strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--quiet");
     ++		strvec_push(&child.args, "--quiet");
      +
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+
     -+	strbuf_release(&refmap);
     -+	return result;
     ++	return !!run_command(&child);
      +}
      +
      +static int fill_each_remote(struct remote *remote, void *cbdata)
     @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
      +		goto cleanup;
      +	}
      +
     -+	/*
     -+	 * Do not modify the result based on the success of the 'fetch'
     -+	 * operation, as a loss of network could cause 'fetch' to fail
     -+	 * quickly. We do not want that to stop the rest of our
     -+	 * background operations.
     -+	 */
      +	for (item = remotes.items;
      +	     item && item < remotes.items + remotes.nr;
      +	     item++)
     -+		fetch_remote(item->string);
     ++		result |= fetch_remote(item->string);
      +
      +cleanup:
      +	string_list_clear(&remotes, 0);
     @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
      +
       static int maintenance_task_gc(void)
       {
     - 	int result;
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	for (i = 0; i < MAX_NUM_TASKS; i++)
     - 		tasks[i] = xcalloc(1, sizeof(struct maintenance_task));
     + 	struct child_process child = CHILD_PROCESS_INIT;
     +@@ builtin/gc.c: struct maintenance_task {
     + };
       
     -+	tasks[num_tasks]->name = "prefetch";
     -+	tasks[num_tasks]->fn = maintenance_task_prefetch;
     -+	num_tasks++;
     -+
     - 	tasks[num_tasks]->name = "gc";
     - 	tasks[num_tasks]->fn = maintenance_task_gc;
     - 	tasks[num_tasks]->enabled = 1;
     + enum maintenance_task_label {
     ++	TASK_PREFETCH,
     + 	TASK_GC,
     + 	TASK_COMMIT_GRAPH,
     + 
     +@@ builtin/gc.c: enum maintenance_task_label {
     + };
     + 
     + static struct maintenance_task tasks[] = {
     ++	[TASK_PREFETCH] = {
     ++		"prefetch",
     ++		maintenance_task_prefetch,
     ++	},
     + 	[TASK_GC] = {
     + 		"gc",
     + 		maintenance_task_gc,
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'run --task duplicate' '
  9:  83648f4865 ! 10:  0128fdfd1a maintenance: add loose-objects task
     @@ Documentation/git-maintenance.txt: gc::
       --auto::
      
       ## builtin/gc.c ##
     -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
     - 	return 0;
     - }
     - 
     --#define MAX_NUM_TASKS 3
     -+#define MAX_NUM_TASKS 4
     - 
     - static const char * const builtin_maintenance_usage[] = {
     - 	N_("git maintenance run [<options>]"),
      @@ builtin/gc.c: static int maintenance_task_gc(void)
     - 	return result;
     + 	return run_command(&child);
       }
       
      +static int prune_packed(void)
      +{
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	argv_array_pushl(&cmd, "prune-packed", NULL);
     ++	struct child_process child = CHILD_PROCESS_INIT;
     ++
     ++	child.git_cmd = 1;
     ++	strvec_push(&child.args, "prune-packed");
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--quiet");
     ++		strvec_push(&child.args, "--quiet");
      +
     -+	return run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     ++	return !!run_command(&child);
      +}
      +
      +struct write_loose_object_data {
     @@ builtin/gc.c: static int maintenance_task_gc(void)
      +	int batch_size;
      +};
      +
     -+static int loose_object_exists(const struct object_id *oid,
     -+			       const char *path,
     -+			       void *data)
     ++static int bail_on_loose(const struct object_id *oid,
     ++			 const char *path,
     ++			 void *data)
      +{
      +	return 1;
      +}
     @@ builtin/gc.c: static int maintenance_task_gc(void)
      +	struct repository *r = the_repository;
      +	int result = 0;
      +	struct write_loose_object_data data;
     -+	struct strbuf prefix = STRBUF_INIT;
     -+	struct child_process *pack_proc;
     ++	struct child_process pack_proc = CHILD_PROCESS_INIT;
      +
      +	/*
      +	 * Do not start pack-objects process
      +	 * if there are no loose objects.
      +	 */
      +	if (!for_each_loose_file_in_objdir(r->objects->odb->path,
     -+					   loose_object_exists,
     ++					   bail_on_loose,
      +					   NULL, NULL, NULL))
      +		return 0;
      +
     -+	pack_proc = xmalloc(sizeof(*pack_proc));
     -+
     -+	child_process_init(pack_proc);
     ++	pack_proc.git_cmd = 1;
      +
     -+	strbuf_addstr(&prefix, r->objects->odb->path);
     -+	strbuf_addstr(&prefix, "/pack/loose");
     -+
     -+	argv_array_pushl(&pack_proc->args, "git", "pack-objects", NULL);
     ++	strvec_push(&pack_proc.args, "pack-objects");
      +	if (opts.quiet)
     -+		argv_array_push(&pack_proc->args, "--quiet");
     -+	argv_array_push(&pack_proc->args, prefix.buf);
     ++		strvec_push(&pack_proc.args, "--quiet");
     ++	strvec_pushf(&pack_proc.args, "%s/pack/loose", r->objects->odb->path);
      +
     -+	pack_proc->in = -1;
     ++	pack_proc.in = -1;
      +
     -+	if (start_command(pack_proc)) {
     ++	if (start_command(&pack_proc)) {
      +		error(_("failed to start 'git pack-objects' process"));
     -+		result = 1;
     -+		goto cleanup;
     ++		return 1;
      +	}
      +
     -+	data.in = xfdopen(pack_proc->in, "w");
     ++	data.in = xfdopen(pack_proc.in, "w");
      +	data.count = 0;
      +	data.batch_size = 50000;
      +
     @@ builtin/gc.c: static int maintenance_task_gc(void)
      +
      +	fclose(data.in);
      +
     -+	if (finish_command(pack_proc)) {
     ++	if (finish_command(&pack_proc)) {
      +		error(_("failed to finish 'git pack-objects' process"));
      +		result = 1;
      +	}
      +
     -+cleanup:
     -+	strbuf_release(&prefix);
     -+	free(pack_proc);
      +	return result;
      +}
      +
     @@ builtin/gc.c: static int maintenance_task_gc(void)
       typedef int maintenance_task_fn(void);
       
       struct maintenance_task {
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	tasks[num_tasks]->fn = maintenance_task_prefetch;
     - 	num_tasks++;
     +@@ builtin/gc.c: struct maintenance_task {
       
     -+	tasks[num_tasks]->name = "loose-objects";
     -+	tasks[num_tasks]->fn = maintenance_task_loose_objects;
     -+	num_tasks++;
     -+
     - 	tasks[num_tasks]->name = "gc";
     - 	tasks[num_tasks]->fn = maintenance_task_gc;
     - 	tasks[num_tasks]->enabled = 1;
     + enum maintenance_task_label {
     + 	TASK_PREFETCH,
     ++	TASK_LOOSE_OBJECTS,
     + 	TASK_GC,
     + 	TASK_COMMIT_GRAPH,
     + 
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 		"prefetch",
     + 		maintenance_task_prefetch,
     + 	},
     ++	[TASK_LOOSE_OBJECTS] = {
     ++		"loose-objects",
     ++		maintenance_task_loose_objects,
     ++	},
     + 	[TASK_GC] = {
     + 		"gc",
     + 		maintenance_task_gc,
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '
     @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '
      +
      +	# Hack to stop maintenance from running during "git commit"
      +	echo in use >.git/objects/maintenance.lock &&
     ++
     ++	# Assuming that "git commit" creates at least one loose object
      +	test_commit create-loose-object &&
      +	rm .git/objects/maintenance.lock &&
      +
     @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '
      +	test_cmp obj-dir-before obj-dir-between &&
      +	ls .git/objects/pack/*.pack >packs-between &&
      +	test_line_count = 2 packs-between &&
     ++	ls .git/objects/pack/loose-*.pack >loose-packs &&
     ++	test_line_count = 1 loose-packs &&
      +
      +	# The second run deletes loose objects
      +	# but does not create a pack-file.
  -:  ---------- > 11:  c2baf6e119 midx: enable core.multiPackIndex by default
 10:  b6328c2106 ! 12:  00f47c4848 maintenance: add incremental-repack task
     @@ Commit message
             size" is calculated by taking the size of the pack-file divided
             by the number of objects in the pack-file and multiplied by the
             number of objects from the multi-pack-index with offset in that
     -       pack-file. The expected size approximats how much data from that
     +       pack-file. The expected size approximates how much data from that
             pack-file will contribute to the resulting pack-file size. The
             intention is that the resulting pack-file will be close in size
             to the provided batch size.
     @@ builtin/gc.c
       
       #define FAILED_RUN "failed to run %s"
       
     -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)
     - 	return 0;
     - }
     - 
     --#define MAX_NUM_TASKS 4
     -+#define MAX_NUM_TASKS 5
     - 
     - static const char * const builtin_maintenance_usage[] = {
     - 	N_("git maintenance run [<options>]"),
      @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
       	return prune_packed() || pack_loose();
       }
       
      +static int multi_pack_index_write(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	argv_array_pushl(&cmd, "multi-pack-index", "write", NULL);
     ++	struct child_process child = CHILD_PROCESS_INIT;
     ++
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "multi-pack-index", "write", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     ++		strvec_push(&child.args, "--no-progress");
      +
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
     ++	if (run_command(&child))
     ++		return error(_("failed to write multi-pack-index"));
      +
     -+	return result;
     ++	return 0;
      +}
      +
      +static int rewrite_multi_pack_index(void)
     @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
      +	unlink(midx_name);
      +	free(midx_name);
      +
     -+	if (multi_pack_index_write()) {
     -+		error(_("failed to rewrite multi-pack-index"));
     -+		return 1;
     -+	}
     -+
     -+	return 0;
     ++	return multi_pack_index_write();
      +}
      +
     -+static int multi_pack_index_verify(void)
     ++static int multi_pack_index_verify(const char *message)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	argv_array_pushl(&cmd, "multi-pack-index", "verify", NULL);
     ++	struct child_process child = CHILD_PROCESS_INIT;
     ++
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "multi-pack-index", "verify", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     ++		strvec_push(&child.args, "--no-progress");
      +
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
     ++	if (run_command(&child)) {
     ++		warning(_("'git multi-pack-index verify' failed %s"), message);
     ++		return 1;
     ++	}
      +
     -+	return result;
     ++	return 0;
      +}
      +
      +static int multi_pack_index_expire(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	argv_array_pushl(&cmd, "multi-pack-index", "expire", NULL);
     ++	struct child_process child = CHILD_PROCESS_INIT;
     ++
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "multi-pack-index", "expire", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     ++		strvec_push(&child.args, "--no-progress");
      +
      +	close_object_store(the_repository->objects);
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	argv_array_clear(&cmd);
      +
     -+	return result;
     ++	if (run_command(&child))
     ++		return error(_("'git multi-pack-index expire' failed"));
     ++
     ++	return 0;
      +}
      +
      +static int multi_pack_index_repack(void)
      +{
     -+	int result;
     -+	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	argv_array_pushl(&cmd, "multi-pack-index", "repack", NULL);
     ++	struct child_process child = CHILD_PROCESS_INIT;
     ++
     ++	child.git_cmd = 1;
     ++	strvec_pushl(&child.args, "multi-pack-index", "repack", NULL);
      +
      +	if (opts.quiet)
     -+		argv_array_push(&cmd, "--no-progress");
     ++		strvec_push(&child.args, "--no-progress");
      +
     -+	argv_array_push(&cmd, "--batch-size=0");
     ++	strvec_push(&child.args, "--batch-size=0");
      +
      +	close_object_store(the_repository->objects);
     -+	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
      +
     -+	if (result && multi_pack_index_verify()) {
     -+		warning(_("multi-pack-index verify failed after repack"));
     -+		result = rewrite_multi_pack_index();
     -+	}
     ++	if (run_command(&child))
     ++		return error(_("'git multi-pack-index repack' failed"));
      +
     -+	return result;
     ++	return 0;
      +}
      +
      +static int maintenance_task_incremental_repack(void)
      +{
     -+	if (multi_pack_index_write()) {
     -+		error(_("failed to write multi-pack-index"));
     -+		return 1;
     ++	prepare_repo_settings(the_repository);
     ++	if (!the_repository->settings.core_multi_pack_index) {
     ++		warning(_("skipping incremental-repack task because core.multiPackIndex is disabled"));
     ++		return 0;
      +	}
      +
     -+	if (multi_pack_index_verify()) {
     -+		warning(_("multi-pack-index verify failed after initial write"));
     -+		return rewrite_multi_pack_index();
     -+	}
     -+
     -+	if (multi_pack_index_expire()) {
     -+		error(_("multi-pack-index expire failed"));
     ++	if (multi_pack_index_write())
      +		return 1;
     -+	}
     -+
     -+	if (multi_pack_index_verify()) {
     -+		warning(_("multi-pack-index verify failed after expire"));
     ++	if (multi_pack_index_verify("after initial write"))
      +		return rewrite_multi_pack_index();
     -+	}
     -+
     -+	if (multi_pack_index_repack()) {
     -+		error(_("multi-pack-index repack failed"));
     ++	if (multi_pack_index_expire())
      +		return 1;
     -+	}
     -+
     ++	if (multi_pack_index_verify("after expire step"))
     ++		return !!rewrite_multi_pack_index();
     ++	if (multi_pack_index_repack())
     ++		return 1;
     ++	if (multi_pack_index_verify("after repack step"))
     ++		return !!rewrite_multi_pack_index();
      +	return 0;
      +}
      +
       typedef int maintenance_task_fn(void);
       
       struct maintenance_task {
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	tasks[num_tasks]->fn = maintenance_task_loose_objects;
     - 	num_tasks++;
     +@@ builtin/gc.c: struct maintenance_task {
     + enum maintenance_task_label {
     + 	TASK_PREFETCH,
     + 	TASK_LOOSE_OBJECTS,
     ++	TASK_INCREMENTAL_REPACK,
     + 	TASK_GC,
     + 	TASK_COMMIT_GRAPH,
       
     -+	tasks[num_tasks]->name = "incremental-repack";
     -+	tasks[num_tasks]->fn = maintenance_task_incremental_repack;
     -+	num_tasks++;
     -+
     - 	tasks[num_tasks]->name = "gc";
     - 	tasks[num_tasks]->fn = maintenance_task_gc;
     - 	tasks[num_tasks]->enabled = 1;
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 		"loose-objects",
     + 		maintenance_task_loose_objects,
     + 	},
     ++	[TASK_INCREMENTAL_REPACK] = {
     ++		"incremental-repack",
     ++		maintenance_task_incremental_repack,
     ++	},
     + 	[TASK_GC] = {
     + 		"gc",
     + 		maintenance_task_gc,
      
       ## midx.c ##
      @@
     @@ midx.h: struct multi_pack_index {
       int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
       int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
      
     + ## t/t5319-multi-pack-index.sh ##
     +@@
     + test_description='multi-pack-indexes'
     + . ./test-lib.sh
     + 
     ++GIT_TEST_MULTI_PACK_INDEX=0
     + objdir=.git/objects
     + 
     + midx_read_expect () {
     +
       ## t/t7900-maintenance.sh ##
     +@@ t/t7900-maintenance.sh: test_description='git maintenance builtin'
     + . ./test-lib.sh
     + 
     + GIT_TEST_COMMIT_GRAPH=0
     ++GIT_TEST_MULTI_PACK_INDEX=0
     + 
     + test_expect_success 'help text' '
     + 	test_expect_code 129 git maintenance -h 2>err &&
      @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '
       	test_cmp packs-between packs-after
       '
 11:  478c7f1d0b ! 13:  ef2a231956 maintenance: auto-size incremental-repack batch
     @@ Commit message
          When repacking during the 'incremental-repack' task, we use the
          --batch-size option in 'git multi-pack-index repack'. The initial setting
          used --batch-size=0 to repack everything into a single pack-file. This is
     -    not sustaintable for a large repository. The amount of work required is
     +    not sustainable for a large repository. The amount of work required is
          also likely to use too many system resources for a background job.
      
          Update the 'incremental-repack' task by dynamically computing a
     @@ Commit message
      
       ## builtin/gc.c ##
      @@ builtin/gc.c: static int multi_pack_index_expire(void)
     - 	return result;
     + 	return 0;
       }
       
     -+#define TWO_GIGABYTES (2147483647)
     -+#define UNSET_BATCH_SIZE ((unsigned long)-1)
     ++#define TWO_GIGABYTES (0x7FFF)
      +
      +static off_t get_auto_pack_size(void)
      +{
     @@ builtin/gc.c: static int multi_pack_index_expire(void)
      +
       static int multi_pack_index_repack(void)
       {
     - 	int result;
     - 	struct argv_array cmd = ARGV_ARRAY_INIT;
     -+	struct strbuf batch_arg = STRBUF_INIT;
     -+
     - 	argv_array_pushl(&cmd, "multi-pack-index", "repack", NULL);
     - 
     + 	struct child_process child = CHILD_PROCESS_INIT;
     +@@ builtin/gc.c: static int multi_pack_index_repack(void)
       	if (opts.quiet)
     - 		argv_array_push(&cmd, "--no-progress");
     + 		strvec_push(&child.args, "--no-progress");
       
     --	argv_array_push(&cmd, "--batch-size=0");
     -+	strbuf_addf(&batch_arg, "--batch-size=%"PRIuMAX,
     -+		    (uintmax_t)get_auto_pack_size());
     -+	argv_array_push(&cmd, batch_arg.buf);
     +-	strvec_push(&child.args, "--batch-size=0");
     ++	strvec_pushf(&child.args, "--batch-size=%"PRIuMAX,
     ++				  (uintmax_t)get_auto_pack_size());
       
       	close_object_store(the_repository->objects);
     - 	result = run_command_v_opt(cmd.argv, RUN_GIT_CMD);
     -+	strbuf_release(&batch_arg);
       
     - 	if (result && multi_pack_index_verify()) {
     - 		warning(_("multi-pack-index verify failed after repack"));
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
 12:  a3c64930a0 ! 14:  99840c4b8f maintenance: create maintenance.<task>.enabled config
     @@ Documentation/git-maintenance.txt: SUBCOMMANDS
      
       ## builtin/gc.c ##
      @@ builtin/gc.c: static int maintenance_run(void)
     - static void initialize_tasks(void)
     - {
     - 	int i;
     -+	struct strbuf config_name = STRBUF_INIT;
     - 	num_tasks = 0;
     + 	return result;
     + }
       
     - 	for (i = 0; i < MAX_NUM_TASKS; i++)
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 	tasks[num_tasks]->name = "commit-graph";
     - 	tasks[num_tasks]->fn = maintenance_task_commit_graph;
     - 	num_tasks++;
     -+
     -+	for (i = 0; i < num_tasks; i++) {
     ++static void initialize_task_config(void)
     ++{
     ++	int i;
     ++	struct strbuf config_name = STRBUF_INIT;
     ++	for (i = 0; i < TASK__COUNT; i++) {
      +		int config_value;
      +
      +		strbuf_setlen(&config_name, 0);
     -+		strbuf_addf(&config_name, "maintenance.%s.enabled", tasks[i]->name);
     ++		strbuf_addf(&config_name, "maintenance.%s.enabled",
     ++			    tasks[i].name);
      +
      +		if (!git_config_get_bool(config_name.buf, &config_value))
     -+			tasks[i]->enabled = config_value;
     ++			tasks[i].enabled = config_value;
      +	}
      +
      +	strbuf_release(&config_name);
     - }
     - 
     ++}
     ++
       static int task_option_parse(const struct option *opt,
     + 			     const char *arg, int unset)
     + {
     +@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)
     + 				   builtin_maintenance_options);
     + 
     + 	opts.quiet = !isatty(2);
     ++	initialize_task_config();
     + 
     + 	argc = parse_options(argc, argv, prefix,
     + 			     builtin_maintenance_options,
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'run [--auto|--quiet]' '
 13:  dbacc2b76c ! 15:  a087c63572 maintenance: use pointers to check --auto
     @@ builtin/gc.c: static int maintenance_task_incremental_repack(void)
       	const char *name;
       	maintenance_task_fn *fn;
      +	maintenance_auto_fn *auto_condition;
     - 	int task_order;
       	unsigned enabled:1,
       		 selected:1;
     + 	int selected_order;
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 	[TASK_GC] = {
     + 		"gc",
     + 		maintenance_task_gc,
     ++		need_to_gc,
     + 		1,
     + 	},
     + 	[TASK_COMMIT_GRAPH] = {
      @@ builtin/gc.c: static int maintenance_run(void)
     - 		if (!opts.tasks_selected && !tasks[i]->enabled)
     + 		if (!opts.tasks_selected && !tasks[i].enabled)
       			continue;
       
      +		if (opts.auto_flag &&
     -+		    (!tasks[i]->auto_condition ||
     -+		     !tasks[i]->auto_condition()))
     ++		    (!tasks[i].auto_condition ||
     ++		     !tasks[i].auto_condition()))
      +			continue;
      +
     - 		result = tasks[i]->fn();
     - 	}
     - 
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 
     - 	tasks[num_tasks]->name = "gc";
     - 	tasks[num_tasks]->fn = maintenance_task_gc;
     -+	tasks[num_tasks]->auto_condition = need_to_gc;
     - 	tasks[num_tasks]->enabled = 1;
     - 	num_tasks++;
     - 
     -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)
     - 				   builtin_maintenance_options);
     - 
     - 	opts.quiet = !isatty(2);
     + 		if (tasks[i].fn()) {
     + 			error(_("task '%s' failed"), tasks[i].name);
     + 			result = 1;
     +@@ builtin/gc.c: static void initialize_task_config(void)
     + {
     + 	int i;
     + 	struct strbuf config_name = STRBUF_INIT;
      +	gc_config();
     - 	initialize_tasks();
     ++
     + 	for (i = 0; i < TASK__COUNT; i++) {
     + 		int config_value;
       
     - 	argc = parse_options(argc, argv, prefix,
      
       ## t/t5514-fetch-multiple.sh ##
      @@ t/t5514-fetch-multiple.sh: test_expect_success 'git fetch --multiple (two remotes)' '
 14:  9af2309f08 ! 16:  ef3a854508 maintenance: add auto condition for commit-graph task
     @@ builtin/gc.c: static struct maintenance_opts {
      +
       static int run_write_commit_graph(void)
       {
     - 	int result;
     -@@ builtin/gc.c: static void initialize_tasks(void)
     + 	struct child_process child = CHILD_PROCESS_INIT;
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 	[TASK_COMMIT_GRAPH] = {
     + 		"commit-graph",
     + 		maintenance_task_commit_graph,
     ++		should_write_commit_graph,
     + 	},
     + };
       
     - 	tasks[num_tasks]->name = "commit-graph";
     - 	tasks[num_tasks]->fn = maintenance_task_commit_graph;
     -+	tasks[num_tasks]->auto_condition = should_write_commit_graph;
     - 	num_tasks++;
     - 
     - 	for (i = 0; i < num_tasks; i++) {
      
       ## object.h ##
      @@ object.h: struct object_array {
 15:  42e316ca58 ! 17:  6ac3a58f2f maintenance: create auto condition for loose-objects
     @@ builtin/gc.c: struct write_loose_object_data {
      +					     NULL, NULL, &count);
      +}
      +
     - static int loose_object_exists(const struct object_id *oid,
     - 			       const char *path,
     - 			       void *data)
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 
     - 	tasks[num_tasks]->name = "loose-objects";
     - 	tasks[num_tasks]->fn = maintenance_task_loose_objects;
     -+	tasks[num_tasks]->auto_condition = loose_object_auto_condition;
     - 	num_tasks++;
     - 
     - 	tasks[num_tasks]->name = "incremental-repack";
     + static int bail_on_loose(const struct object_id *oid,
     + 			 const char *path,
     + 			 void *data)
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 	[TASK_LOOSE_OBJECTS] = {
     + 		"loose-objects",
     + 		maintenance_task_loose_objects,
     ++		loose_object_auto_condition,
     + 	},
     + 	[TASK_INCREMENTAL_REPACK] = {
     + 		"incremental-repack",
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '
 16:  3d527cb0dd ! 18:  801b262d1c maintenance: add incremental-repack auto condition
     @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
      +
       static int multi_pack_index_write(void)
       {
     - 	int result;
     -@@ builtin/gc.c: static void initialize_tasks(void)
     - 
     - 	tasks[num_tasks]->name = "incremental-repack";
     - 	tasks[num_tasks]->fn = maintenance_task_incremental_repack;
     -+	tasks[num_tasks]->auto_condition = incremental_repack_auto_condition;
     - 	num_tasks++;
     - 
     - 	tasks[num_tasks]->name = "gc";
     + 	struct child_process child = CHILD_PROCESS_INIT;
     +@@ builtin/gc.c: static struct maintenance_task tasks[] = {
     + 	[TASK_INCREMENTAL_REPACK] = {
     + 		"incremental-repack",
     + 		maintenance_task_incremental_repack,
     ++		incremental_repack_auto_condition,
     + 	},
     + 	[TASK_GC] = {
     + 		"gc",
      
       ## t/t7900-maintenance.sh ##
      @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
 17:  a0f00f8ab8 = 19:  9b4cef7635 midx: use start_delayed_progress()
 18:  f24db7739f ! 20:  39eb83ad1e maintenance: add trace2 regions for task execution
     @@ Commit message
      
       ## builtin/gc.c ##
      @@ builtin/gc.c: static int maintenance_run(void)
     - 		     !tasks[i]->auto_condition()))
     + 		     !tasks[i].auto_condition()))
       			continue;
       
     -+		trace2_region_enter("maintenance", tasks[i]->name, r);
     - 		result = tasks[i]->fn();
     -+		trace2_region_leave("maintenance", tasks[i]->name, r);
     ++		trace2_region_enter("maintenance", tasks[i].name, r);
     + 		if (tasks[i].fn()) {
     + 			error(_("task '%s' failed"), tasks[i].name);
     + 			result = 1;
     + 		}
     ++		trace2_region_leave("maintenance", tasks[i].name, r);
       	}
       
       	rollback_lock_file(&lk);

-- 
gitgitgadget




[Index of Archives]     [Linux Kernel Development]     [Gcc Help]     [IETF Annouce]     [DCCP]     [Netdev]     [Networking]     [Security]     [V4L]     [Bugtraq]     [Yosemite]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux SCSI]     [Fedora Users]

  Powered by Linux