Re: [bug] git svn fetch: defect history - missing merges and wrong tag ancestors - third-party tool found

["M. Buecher" <maddes+git@xxxxxxxxxx>]

On 2021-01-03 19:12, M. Buecher wrote:
> On 2020-12-25 23:24, M. Buecher wrote:
> > Dear all,
> >
> > I finally had the time to start converting some older Subversion
> > repositories to git repositiers and run into issues with repos using
> > "parallel" branches, so-called vendor branches [1] in Subversion to
> > track upstream changes via snapshots in a separate branch and merge
> > them into the custom build on trunk.
> >
> > Just wanting to convert the Subversion history as-is to git.
> > I studied the git-svn reference docs [2] plus the related chapter of
> > the ProGit book [3] and assume that I understood how git-svn works.
> > Still I'm not a git expert, just a sporadic user.
> >
> > Somehow `git svn fetch` (2.29.2.windows.3) looses merge information
> > between vendor branch and trunk plus tags are referencing the
> > predecessors instead of the original ancestor.
> > Maybe there is a manual way to fix this for that small repository, but
> > it wouldn't be feasible for larger repositories, that's why I deceided
> > to write this bug report.
> > Is there anything I missed? (see my procedure below)
> >
> > Fortunately I ran quite early into these issues and also with a very
> > small repository of just 11 commits, so I can provide a small
> > reproduction case (see below after the links).
> > Hoping you can enhance `git svn fetch`.
>
> Tested further with re-created Subversion repositories, that either
> looked the same but made sure that svn:mergeinfo is present
> (Subversion >=1.5 via "cherry pick merge", no "2-URL merge"), or that
> has the vendor branch being copied from trunk (both attached).
> Only when the vendor branch was a copy from trunk, then svn-git got
> the merges correct. Otherwise - even with svn:mergeinfo - it does not
> get the merges.
>
> Assumption:
> It seems that git svn handles trunk and branches in a special way, but
> Subversion actually does not have branches.
> In Subversion there are just directories and files, and a branch/tag
> is just a copy of another directory+revision and merges can happen
> between any directories independent if they have related ancestry or
> not.
>
> Workaround:
> As git svn does not recognize all merges correctly (especially
> cross-branch copies) those lost links must be added manually.
> I wrote a small GNU awk script (attached) to determine all
> cross-branch copies from an `svnadmin dump`. This way the first
> revision when something got copied is known.
> Running git svn just to that revision, then fixing the parents and
> continuing with git svn up the next revision.
> This helps git svn to find the correct ancestors and correctly build
> the follow merges.
> The parents can be changed with `git replace -f --graft <commit>
> <existing correct parents> <additional correct parents>`.
> Additionally before continuing with git svn this `git replace` change
> can be made permanent with `git-filter-repo --force --replace-refs
> delete-no-add`.
>
>
> > Any help is appreciated, thanks in advance
> > Matthias Bücher

Found a nice tool called reposurgeon [4] that helps a lot to manage 
converting a repository from Subversion to git, even in a scriptable 
way.
The authors also helped out to determine that my repository has some 
awkward commits due to older Subversion versions.

[4] https://gitlab.com/esr/reposurgeon

> > [Links]
> > [1]
> > http://svnbook.red-bean.com/en/1.8/svn.advanced.vendorbr.html#svn.advanced.vendorbr.mirrored-sources
> > [2] https://git-scm.com/docs/git-svn
> > [3] 
> > https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git
> >
> >
> > [System Info]
> > git version:
> > git version 2.29.2.windows.3
> > cpu: x86_64
> > built from commit: d054eb1fc46ff23e7c95756a7c747e2f2864b478
> > sizeof-long: 4
> > sizeof-size_t: 8
> > shell-path: /bin/sh
> > uname: Windows 10.0 19042
> > compiler info: gnuc: 10.2
> > libc info: no libc information available
> > $SHELL (typically, interactive shell): C:\Program 
> > Files\Git\usr\bin\bash.exe
> >
> > [Enabled Hooks]
> > none
> >
> >
> > [Commits in Subversion]
> > A Subversion repository dump is attached, plus a test where I
> > recreated the same history directly in a git repository without an
> > issue.
> >
> > * trunk:  a------------d--e--h--i--j--k
> > *                     /     /
> > * vendor: a (empty)--b-----f
> > *                    ^     ^
> > * tags:              c     g
> >
> > * Vendor releases: b, f
> > * Custom modifications: e, i, j, k
> > * Tags: really just used as tags, although Subversion internally they
> > are branches. Therefore wanting to create lightweight git tags,
> > although annotated git tags would be fine too.
> >
> >
> > [Expected Subversion to git repo conversion]
> > * trunk => branch "main"
> > * branches/* => branch "*"
> > * tags/* => tag "*"
> > * vendor/current => branch "vendor/current" (can be renamed later to
> > just "vendor")
> > * vendor/* => tag "vendor/*" (except for vendor/current)
> >
> > Expected "tags":
> > vendor/5.4 = rev b (maybe c when annotated)
> > vendor/5.8 = rev f (maybe g when annotated)
> >
> >
> > [Wrong Results]
> > Merges from "vendor/current" branch to trunk get lost.
> > Tags are referencing the predecessors of the expected commit.
> >
> >
> > [Procedure]
> > ```
> > ### a) preparation
> > cd /d/Coding
> > #
> > cd dd-formmailer
> > svn log --xml --quiet | grep author | sort -u | perl -pe
> > 's/.*>(.*?)<.*/$1 = /' > authors-transform.txt
> > ## edit authors-transform.txt accordingly
> >
> > ### b) git-svn adapted from ProGit book, but with "svn/" prefix
> > cd /d/Coding
> > git svn init --stdlayout --no-metadata --prefix="svn/" --
> > 'svn+ssh://svn@vcs/dd-formmailer' dd-formmailer-git
> > cd dd-formmailer-git
> > ## edit .git/config for additional vendor branches and tags
> > << __EOF
> > ...
> > [svn-remote "svn"]
> > 	...
> > 	fetch = trunk:refs/remotes/svn/trunk
> > #	branches = vendor/current:refs/remotes/svn/vendor/current ##
> > non-glob definition not working for branches
> > 	branches = vendor/{current}:refs/remotes/svn/vendor/*
> > 	branches = branches/*:refs/remotes/svn/*
> > 	tags = vendor/*:refs/remotes/svn/tags/vendor/*
> > 	tags = tags/*:refs/remotes/svn/tags/*
> > __EOF
> > ##
> > cat .git/config
> > git svn fetch --authors-file ../dd-formmailer/authors-transform.txt
> > #
> > git branch -vv --list ; git for-each-ref
> > gitk --all &
> > ```
> >
> > [Log]
> > $ cat .git/config
> > [core]
> >         repositoryformatversion = 0
> >         filemode = false
> >         bare = false
> >         logallrefupdates = true
> >         symlinks = false
> >         ignorecase = true
> > [svn-remote "svn"]
> >         noMetadata = 1
> >         url = svn+ssh://svn@vcs/dd-formmailer
> >         fetch = trunk:refs/remotes/svn/trunk
> >         branches = vendor/{current}:refs/remotes/svn/vendor/*
> >         branches = branches/*:refs/remotes/svn/*
> >         tags = vendor/*:refs/remotes/svn/tags/vendor/*
> >         tags = tags/*:refs/remotes/svn/tags/*
> >
> > $ git svn fetch --authors-file ../dd-formmailer/authors-transform.txt
> > r1 = 158d53044a5628897379403647a19ea13594b532 
> > (refs/remotes/svn/vendor/current)
> >         A       _svn__guideline.txt
> >         A       _svn_client_config.txt
> >         A       _svn_dir_ignore_list.txt
> > r1 = a795b654edbc296e6f38da398f032a4851fd0a9e (refs/remotes/svn/trunk)
> >         A       dd-formmailer.css
> >         A       dd-formmailer.php
> >         A       lang/BrazilianPortuguese.php
> >         A       lang/Catalan.php
> >         A       lang/Danish.php
> >         A       lang/Deutsch.php
> >         A       lang/Dutch.php
> >         A       lang/English.php
> >         A       lang/Finnish.php
> >         A       lang/French.php
> >         A       lang/Greek.php
> >         A       lang/Italian.php
> >         A       lang/NorwegianBokmaal.php
> >         A       lang/Polish.php
> >         A       lang/Portuguese.php
> >         A       lang/Romanian.php
> >         A       lang/Russian.php
> >         A       lang/Slovak.php
> >         A       lang/Slovene.php
> >         A       lang/Spanish.php
> >         A       lang/Swedish.php
> >         A       lang/Turkish.php
> >         A       recaptchalib.php
> > r2 = 8fd8668dbed2dde6b55306c71b3b629f5ed794ec 
> > (refs/remotes/svn/vendor/current)
> > Found possible branch point:
> > svn+ssh://svn@vcs/dd-formmailer/vendor/current =>
> > svn+ssh://svn@vcs/dd-formmailer/vendor/5.4, 1
> > Found branch parent: (refs/remotes/svn/tags/vendor/5.4)
> > 158d53044a5628897379403647a19ea13594b532
> > Following parent with do_switch
> >         A       dd-formmailer.css
> >         A       dd-formmailer.php
> >         A       lang/BrazilianPortuguese.php
> >         A       lang/Catalan.php
> >         A       lang/Danish.php
> >         A       lang/Deutsch.php
> >         A       lang/Dutch.php
> >         A       lang/English.php
> >         A       lang/Finnish.php
> >         A       lang/French.php
> >         A       lang/Greek.php
> >         A       lang/Italian.php
> >         A       lang/NorwegianBokmaal.php
> >         A       lang/Polish.php
> >         A       lang/Portuguese.php
> >         A       lang/Romanian.php
> >         A       lang/Russian.php
> >         A       lang/Slovak.php
> >         A       lang/Slovene.php
> >         A       lang/Spanish.php
> >         A       lang/Swedish.php
> >         A       lang/Turkish.php
> >         A       recaptchalib.php
> > Successfully followed parent
> > r3 = 7ab4f436cafc8af3ed2e727a6d8cbef1a8f8b39f 
> > (refs/remotes/svn/tags/vendor/5.4)
> >         A       dd-formmailer.css
> >         A       dd-formmailer.php
> >         A       lang/BrazilianPortuguese.php
> >         A       lang/Catalan.php
> >         A       lang/Danish.php
> >         A       lang/Deutsch.php
> >         A       lang/Dutch.php
> >         A       lang/English.php
> >         A       lang/Finnish.php
> >         A       lang/French.php
> >         A       lang/Greek.php
> >         A       lang/Italian.php
> >         A       lang/NorwegianBokmaal.php
> >         A       lang/Polish.php
> >         A       lang/Portuguese.php
> >         A       lang/Romanian.php
> >         A       lang/Russian.php
> >         A       lang/Slovak.php
> >         A       lang/Slovene.php
> >         A       lang/Spanish.php
> >         A       lang/Swedish.php
> >         A       lang/Turkish.php
> >         A       recaptchalib.php
> > r4 = 7ec3663fd8e7c9fcfeb2742968a948d6978776d4 (refs/remotes/svn/trunk)
> >         M       dd-formmailer.css
> >         M       dd-formmailer.php
> >         A       dd-verify.php
> > r5 = e65ba2bdcd12a1935c1a327507dad6b7117f452b (refs/remotes/svn/trunk)
> >         A       calendar.gif
> >         A       date_chooser.js
> >         M       dd-formmailer.css
> >         M       dd-formmailer.php
> >         A       lang/Belarussian.php
> >         A       lang/Czech.php
> >         A       lang/Estonian.php
> >         A       lang/Japanese.php
> >         A       lang/Vietnamese.php
> >         M       recaptchalib.php
> > r6 = 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc 
> > (refs/remotes/svn/vendor/current)
> > Found possible branch point:
> > svn+ssh://svn@vcs/dd-formmailer/vendor/current =>
> > svn+ssh://svn@vcs/dd-formmailer/vendor/5.8, 3
> > Found branch parent: (refs/remotes/svn/tags/vendor/5.8)
> > 8fd8668dbed2dde6b55306c71b3b629f5ed794ec
> > Following parent with do_switch
> >         A       calendar.gif
> >         A       date_chooser.js
> >         M       dd-formmailer.css
> >         M       dd-formmailer.php
> >         A       lang/Belarussian.php
> >         A       lang/Czech.php
> >         A       lang/Estonian.php
> >         A       lang/Japanese.php
> >         A       lang/Vietnamese.php
> >         M       recaptchalib.php
> > Successfully followed parent
> > r7 = cc00cb187386298cf974dda69f151d2ad4795917 
> > (refs/remotes/svn/tags/vendor/5.8)
> >         A       calendar.gif
> >         A       date_chooser.js
> >         M       dd-formmailer.css
> >         M       dd-formmailer.php
> >         M       dd-verify.php
> >         A       lang/Belarussian.php
> >         A       lang/Czech.php
> >         A       lang/Estonian.php
> >         A       lang/Japanese.php
> >         A       lang/Vietnamese.php
> >         M       recaptchalib.php
> > Checking svn:mergeinfo changes since r5: 1 sources, 1 changed
> > W: Cannot find common ancestor between
> > e65ba2bdcd12a1935c1a327507dad6b7117f452b and
> > 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc. Ignoring merge info.
> > r8 = 9a5cd8f55f377f469280171cd87219b8f528c693 (refs/remotes/svn/trunk)
> >         M       dd-formmailer.php
> > r9 = 8143782376d9fbc91a9181b367b72701205b017f (refs/remotes/svn/trunk)
> >         M       dd-formmailer.php
> > r10 = 6f563fc4c511ddcc53c2ffc5d26c1e116725bc6b 
> > (refs/remotes/svn/trunk)
> >         M       _svn_client_config.txt
> > r11 = e9130854178d2c2743a981f303cdd7f34e54b052 
> > (refs/remotes/svn/trunk)
> > svnserve: E210002: Network connection closed unexpectedly
> > Checked out HEAD:
> >   svn+ssh://svn@vcs/dd-formmailer/trunk r11
> >
> > $ git branch -vv --list ; git for-each-ref
> > * main e913085 Updated client config for Windows Scripts
> > e9130854178d2c2743a981f303cdd7f34e54b052 commit refs/heads/main
> > 7ab4f436cafc8af3ed2e727a6d8cbef1a8f8b39f commit 
> > refs/remotes/svn/tags/vendor/5.4
> > cc00cb187386298cf974dda69f151d2ad4795917 commit 
> > refs/remotes/svn/tags/vendor/5.8
> > e9130854178d2c2743a981f303cdd7f34e54b052 commit refs/remotes/svn/trunk
> > 9f95df7ce49c5d1cb4715017d223a8bd1c8dcffc commit 
> > refs/remotes/svn/vendor/current