This replaces origin/sb/submodule-parallel-update
(anchoring at 74367d8938, Merge branch 'sb/submodule-parallel-fetch'
into sb/submodule-parallel-update)
What does it do?
---
This series should finish the on going efforts of parallelizing
submodule network traffic. The patches contain tests for clone,
fetch and submodule update to use the actual parallelism both via
command line as well as a configured option. I decided to go with
"submodule.jobs" for all three for now.
What is new in v2?
---
* The patches got reordered slightly
* Documentation was adapted
Interdiff below
Stefan Beller (8):
run_processes_parallel: Add output to tracing messages
submodule config: keep update strategy around
submodule config: remove name_and_item_from_var
submodule-config: parse_config
fetching submodules: Respect `submodule.jobs` config option
git submodule update: have a dedicated helper for cloning
submodule update: expose parallelism to the user
clone: allow an explicit argument for parallel submodule clones
Documentation/config.txt | 7 ++
Documentation/git-clone.txt | 6 +-
Documentation/git-submodule.txt | 7 +-
builtin/clone.c | 23 +++-
builtin/fetch.c | 2 +-
builtin/submodule--helper.c | 244 ++++++++++++++++++++++++++++++++++++++++
git-submodule.sh | 54 ++++-----
run-command.c | 4 +
submodule-config.c | 98 ++++++++++------
submodule-config.h | 3 +
submodule.c | 5 +
t/t5526-fetch-submodules.sh | 14 +++
t/t7400-submodule-basic.sh | 4 +-
t/t7406-submodule-update.sh | 27 +++++
14 files changed, 418 insertions(+), 80 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 0de0138..785721a 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2643,12 +2643,12 @@ submodule.<name>.ignore::
"--ignore-submodules" option. The 'git submodule' commands are not
affected by this setting.
-submodule::jobs
+submodule.jobs::
This is used to determine how many submodules can be operated on in
parallel. Specifying a positive integer allows up to that number
- of submodules being fetched in parallel. Specifying 0 the number
- of cpus will be taken as the maximum number. Currently this is
- used in fetch and clone operations only.
+ of submodules being fetched in parallel. This is used in fetch
+ and clone operations only. A value of 0 will give some reasonable
+ default. The defaults may change with different versions of Git.
tag.sort::
This variable controls the sort ordering of tags when displayed by
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index affa52e..01bd6b7 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -216,9 +216,10 @@ objects from the source repository into a pack in the cloned repository.
The result is Git repository can be separated from working
tree.
--j::
---jobs::
+-j <n>::
+--jobs <n>::
The number of submodules fetched at the same time.
+ Defaults to the `submodule.jobs` option.
<repository>::
The (possibly remote) repository to clone from. See the
diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index f5429fa..c70fafd 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -374,10 +374,11 @@ for linkgit:git-clone[1]'s `--reference` and `--shared` options carefully.
clone with a history truncated to the specified number of revisions.
See linkgit:git-clone[1]
--j::
---jobs::
+-j <n>::
+--jobs <n>::
This option is only valid for the update command.
Clone new submodules in parallel with as many jobs.
+ Defaults to the `submodule.jobs` option.
<path>...::
Paths to submodule(s). When specified this will restrict the command
diff --git a/builtin/clone.c b/builtin/clone.c
index 5ac2d89..22b9924 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -727,10 +727,7 @@ static int checkout(void)
struct argv_array args = ARGV_ARRAY_INIT;
argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
- if (max_jobs == -1)
- if (git_config_get_int("submodule.jobs", &max_jobs))
- max_jobs = 1;
- if (max_jobs != 1) {
+ if (max_jobs != -1) {
struct strbuf sb = STRBUF_INIT;
strbuf_addf(&sb, "--jobs=%d", max_jobs);
argv_array_push(&args, sb.buf);
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index c3d438a..67dba1c 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -476,9 +476,10 @@ static int update_clone(int argc, const char **argv, const char *prefix)
/* Overlay the parsed .gitmodules file with .git/config */
git_config(git_submodule_config, NULL);
- if (max_jobs == -1)
- if (git_config_get_int("submodule.jobs", &max_jobs))
- max_jobs = 1;
+ if (max_jobs < 0)
+ max_jobs = config_parallel_submodules();
+ if (max_jobs < 0)
+ max_jobs = 1;
run_processes_parallel(max_jobs,
update_clone_get_next_task,
--
2.5.0.281.g4ed9cdb
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html