On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:
> You are correct that git-fast-import is killed by the OOM killer, but I
> was unclear about which process was malloc()ing so much memory that the
> OOM killer got invoked (as other completely unrelated processes usually
> also get killed when this happens).
>
> Unless there's one gigantic file in one change that gets removed by
> another change, I don't think that's the problem; as I mentioned in
> another email, the machine has 32GB physical memory and the largest
> single file in the current head is only 118MB. Even if there is a very
> large transient file somewhere in the history, I seriously doubt it's
> tens of gigabytes in size.
>
> I have tried watching it with top before, but it takes several hours
> before it dies. I haven't been able to see any explosion of memory
> usage, even within the final hour, but I've never caught it just before
> it dies, either. I suspect that whatever the issue is here, it happens
> very quickly.
>
> If I'm unable to get through this today using the incremental p4 sync
> method you described, I'll try running a full-blown clone overnight with
> top in batch mode writing to a log file to see whether it catches
> anything.
>
> Thanks again,
> Corey
Unforunately I have not made much progress. The incremental sync method
fails with the output pasted below. The change I specified is only one
change number above where that repo was cloned...
So I tried a 'git p4 rebase' overnight with top running, and as I feared
I did not see anything out of the ordinary. git, git-fast-import, and
git-p4 all hovered under 1.5% MEM the entire time, right up until
death. The last entry in my log shows git-fast-import at 0.8%, with git
and git-p4 at 0.0% and 0.1%, respectively. I could try again with a
more granular period, but I feel like this method is ultimately a goose
chase.
Corey
$ git p4 sync //path/to/some/branch@505859
Doing initial import of //path/to/some/branch/ from revision @505859 into refs/remotes/p4/master
fast-import failed: warning: Not updating refs/remotes/p4/master (new tip 29ef6ff25f1448fa2f907d22fd704594dc8769bd does not contain d477672be5ac6a00cc9175ba2713d5395660e840)
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 165000
Total objects: 69 ( 232434 duplicates )
blobs : 45 ( 209904 duplicates 40 deltas of 42 attempts)
trees : 23 ( 22530 duplicates 0 deltas of 23 attempts)
commits: 1 ( 0 duplicates 0 deltas of 0 attempts)
tags : 0 ( 0 duplicates 0 deltas of 0 attempts)
Total branches: 1 ( 1 loads )
marks: 1024 ( 0 unique )
atoms: 105170
Memory total: 24421 KiB
pools: 17976 KiB
objects: 6445 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 33554432
pack_report: core.packedGitLimit = 268435456
pack_report: pack_used_ctr = 4371
pack_report: pack_mmap_calls = 124
pack_report: pack_open_windows = 8 / 9
pack_report: pack_mapped = 268435456 / 268435456
---------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html