Re: Importing Bzr revisions

David Reitter <david.reitter@xxxxxxxxx> · Wed, 25 Mar 2009 23:05:20 -0400

On Mar 23, 2009, at 10:07 AM, Teemu Likonen wrote:

I'm just experimenting with "bzr fast-export", which converts to git,
and it seems to take about 4 minutes for 1000 revisions on our
(modern) server. That would be around 7 hours for my emacs  
repository;
I can't do that daily.

I wonder if there's a way for (bzr) fast-export / (git) fast-import  
to
work incrementally, i.e. for selected or most recent revisions.

They can or should work incrementally, and actually I have succesfully
done that. The idea is to use --import-marks and --export-marks  
options
with "git fast-import" and --marks option with "bzr fast-export.

I noticed some problems with newer versions of "bzr fast-export",  
though
(since it was converted to a proper Bzr command). It seems to corrupt
the marks file when doing the first incremental export after the  
initial
export. At least the revisions are not in right order in the marks  
file
anymore. "git fast-import" can't continue to import from the same
revision where it left last time and it seems to create alternative
history -- or something.

Really I don't know if this is a bug in Bzr or in Git and haven't
figured out how to file a useful bug report.

I'm experiencing pretty much the same problem.

Looking at the code (marks_file.py) I don't see why the order would  
matter (even though it would be nicer if the order was consistent).  I  
actually changed this so that it's always sorted, just to help me debug.

Now, I'm getting these errors back from git:

fatal: mark :96985 not declared
fast-import: dumping crash report to .git/fast_import_crash_74262
bzr: broken pipe

I couldn't reproduce this with a simple repository.

However, if one inspects the output of bzr fast-export, one finds  
stuff like this:

commit refs/heads/master
mark :96984
committer <dann> 1237847130 +0000
data 205
....
from :96985
M 644 inline lisp/ChangeLog
data 741178
--- ... --- ... --- ... ---
commit refs/heads/master
mark :96985
committer <jhd> 1237849747 +0000
...
from :96984
M 644 inline src/gtkutil.c
data 135796

I'm not sure about the structure of these files, but my educated guess  
would be that this is a circular reference.
Strange.  One would think that this should never happen.

That said, I get the same errors in other cases as well without  
circular reference.

As an experiment, I deleted the last 1000 or so revisions from the bzr  
marks file, so that they would be output again.
A couple of minutes and 430MB in output later, I imported this on the  
git side, which, after a few seconds, came back with this:
---------------------------------------------------------------------
Alloc'd objects:     105000
Total objects:            5 (      5063 duplicates                  )
      blobs  :            0 (      1549 duplicates          0 deltas)
      trees  :            2 (      2505 duplicates          0 deltas)
      commits:            3 (      1009 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:           1 (         1 loads     )
      marks:        1048576 (     97012 unique    )
      atoms:           1937
Memory total:          5380 KiB
       pools:          2098 KiB
     objects:          3281 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize =   33554432
pack_report: core.packedGitLimit      =  268435456
pack_report: pack_used_ctr            =     104764
pack_report: pack_mmap_calls          =          6
pack_report: pack_open_windows        =          4 /          4
pack_report: pack_mapped              =  100666262 /  100666262
---------------------------------------------------------------------

So, it seems like there were 3 commits that were missing from previous  
transfers.  The final lines of the marks files for bzr and git seem  
coherent.  Hard to identify the culprit.

Further changes resulted in good conversion so far.  Deleting a good  
number of the most recent marks entries seems to be the right thing to  
recover.

A "bzr uncommit" seemed not to make its way to the git side.  No  
good.   I wonder if that is going to create a lasting inconsistency. I  
did get this subsequently:
warning: Not updating refs/heads/master (new tip  
6c81ccc916026d020eadeb0ad6e5b12c18aeccd3 does not contain  
3222cfee5bc00412b6f5e52a420f93564f586ee9)
This "not contained" revision resolved to the uncommitted one - that  
makes sense.
But what consequences does the warning have...?

<<attachment: smime.p7s>>