On Wed, Sep 09 2020, Emily Shaffer wrote:
First, thanks a lot for working on this. As you may have found I've done
some small amount of actual work in this area before, but mostly just
blathered about it on the ML.
> Begin a design document for config-based hooks, managed via git-hook.
> Focus on an overview of the implementation and motivation for design
> decisions. Briefly discuss the alternatives considered before this
> point. Also, attempt to redefine terms to fit into a multihook world.
> [...]
> +[[status-quo]]
> +=== Status quo
> +
> +Today users can implement multihooks themselves by using a "trampoline script"
> +as their hook, and pointing that script to a directory or list of other scripts
> +they wish to run.
...or by setting core.hooksPath in their local/global/system
config. Granted it doesn't cover the malicious hook injection case
you're also trying to solve, but does address e.g. having a git server
with a lot of centralized hooks.
The "trampoline script" also isn't needed for the common case you
mention, you just symlink the .git/hooks directory (as e.g. GitLab
does). People usually use a trampoline script for e.g. using GNU
parallel or something to execute N hooks.
> +[[hook-directories]]
> +=== Hook directories
> +
> +Other contributors have suggested Git learn about the existence of a directory
> +such as `.git/hooks/<hookname>.d` and execute those hooks in alphabetical order.
...which seems like an easy thing to add later by having a "hookdir" in
addition to "hookcmd", i.e. just specify a glob there instead of a
cmd/path.
You already use "hookdir" for something else though, so that's a bit
confusing, perhaps s/hookcmd/definehookcmd/ would be less confusing, or
perhaps more confusing...
> [...]
> +[[execution-ordering]]
> +=== Execution ordering
> +
> +We may find that config order is insufficient for some users; for example,
> +config order makes it difficult to add a new hook to the system or global config
> +which runs at the end of the hook list. A new ordering schema should be:
> +
> +1) Specified by a `hook.order` config, so that users will not unexpectedly see
> +their order change;
> +
> +2) Either dependency or numerically based.
> +
> +Dependency-based ordering is prone to classic linked-list problems, like a
> +cycles and handling of missing dependencies. But, it paves the way for enabling
> +parallelization if some tasks truly depend on others.
>
> +Numerical ordering makes it tricky for Git to generate suggested ordering
> +numbers for each command, but is easy to determine a definitive order.
> +
> +[[parallelization]]
> +=== Parallelization
> +
> +Users with many hooks might want to run them simultaneously, if the hooks don't
> +modify state; if one hook depends on another's output, then users will want to
> +specify those dependencies. If we decide to solve this problem, we may want to
> +look to modern build systems for inspiration on how to manage dependencies and
> +parallel tasks.
If you're taking requests it would make me very happy if we had
parallelism in this from day one. It's the kind of thing that's hard to
do by default once a feature is shipped since people will implicitly
depend on it not being there, i.e. we won't know what we're breaking.
I think doing it this way is simple, covers most use cases, and solves a
lot of the problems you note:
1. Don't use config order to execute hooks, use glob'd name order
regardless of origin. I.e. a system-level hook is called "001-first"
is executed before a local hook called "999-at-the-end" (or the other
way around, i.e. hook origin doesn't matter).
2. We execute hooks parallel in that glob order, i.e. a pthread for-loop
that starts the 001-first task first, eventually getting to
999-at-the-end N at a time. I.e. the same as:
parallel --jobs N --halt-on-error soon,fail=1" ::: <hooks-in-glob-order>
This allows for parallelism but guarantees the very useful case of
having a global log hook being guaranteed to execute.
3. A hook can define "parallel=no" in its config. We'll then run it
while no other hook is running.
4. We don't attempt to do dependencies etc, if you need that sort of
complexity you can just make one of the hooks be a hook runner as
users do now for the common "make it parallel" case.
It's a relatively small change to the code you have already. I.e. the
for_each() in run_hooks() would be called N times for each continuous
glob'd parallel/non-parallel segment, and hook_list()'s config parsing
would learn to spew those out as a list-of-lists.
This also gives you a rudimentary implementation of the dependency
schema you proposed for free. I.e. a definition of (pseudocode):
hookcmd=000-first
parallel=no
hookcmd=250-middle-abc
hookcmd=250-middle-xyz
hookcmd=300-gather
parallel=no
hookcmd=999-the-end
Would result in the pseudocode execution of;
segments=[[000-first],
[250-middle-abc, 250-middle-xyz],
[300-gather],
[999-the-end]]
for each s in segments:
ok = run_in_parallel(s)
last if !ok # or, depending on "early exit?" config
I.e.:
* The common case of people adding N hooks won't take sum(N) time.
* parallel=no hooks aren't run in parallel with other non-parallel
hooks
* We support a rudimentary dependency schema as a side-effect,
i.e. defining 300-gather as non-parallel allows it to act as the sole
"reduce" step in a map/reduce in a "map" step started with the 250-*
hooks.
> +[[securing-hookdir-hooks]]
> +=== Securing hookdir hooks
> +
> +With the design as written in this doc, it's still possible for a malicious user
> +to modify `.git/config` to include `hook.pre-receive.command = rm -rf /`, then
> +zip their repo and send it to another user. It may be necessary to teach Git to
> +only allow inlined hooks like this if they were configured outside of the local
> +scope (in other words, only run hookcmds, and only allow hookcmds to be
> +configured in global or system scope); or another approach, like a list of safe
> +projects, might be useful. It may also be sufficient (or at least useful) to
> +teach a `hook.disableAll` config or similar flag to the Git executable.
I think this part of the doc should note a bit of the context in
https://lore.kernel.org/git/20171002234517.GV19555@xxxxxxxxxxxxxxxxxxxxxxxxx/
I.e. even if we get a 100% secure hook implementation we've done
practically nothing for overall security, since we'll still run the
pager, aliases etc. from that local repo.
This is a great step in the right direction, but it behooves us to note
that, so some user reading this documentation without context doesn't
think inspecting untrusted repositories like that is safe just because
they set the right hook settings in their config (once what's being
proposed here is implemented).
