I guess you could try changing the OOM score for git-fast-import.
change /proc/<pid>/oomadj.
I think a value of -31 would make it very unlikely to be killed.
On 29/08/13 23:46, Pete Wyckoff wrote:
cmtptr@xxxxxxxxx wrote on Wed, 28 Aug 2013 11:41 -0400:
On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:
You are correct that git-fast-import is killed by the OOM killer, but I
was unclear about which process was malloc()ing so much memory that the
OOM killer got invoked (as other completely unrelated processes usually
also get killed when this happens).
Unless there's one gigantic file in one change that gets removed by
another change, I don't think that's the problem; as I mentioned in
another email, the machine has 32GB physical memory and the largest
single file in the current head is only 118MB. Even if there is a very
large transient file somewhere in the history, I seriously doubt it's
tens of gigabytes in size.
I have tried watching it with top before, but it takes several hours
before it dies. I haven't been able to see any explosion of memory
usage, even within the final hour, but I've never caught it just before
it dies, either. I suspect that whatever the issue is here, it happens
very quickly.
If I'm unable to get through this today using the incremental p4 sync
method you described, I'll try running a full-blown clone overnight with
top in batch mode writing to a log file to see whether it catches
anything.
Thanks again,
Corey
Unforunately I have not made much progress. The incremental sync method
fails with the output pasted below. The change I specified is only one
change number above where that repo was cloned...
I usually just do "git p4 sync @505859". The error message below
crops up when things get confused. Usually after a previous
error. I tend to destroy the repo and try again. Sorry I don't
can't explain better what's happening here. It's not a memory
issue; it reports only 24 MB used.
So I tried a 'git p4 rebase' overnight with top running, and as I feared
I did not see anything out of the ordinary. git, git-fast-import, and
git-p4 all hovered under 1.5% MEM the entire time, right up until
death. The last entry in my log shows git-fast-import at 0.8%, with git
and git-p4 at 0.0% and 0.1%, respectively. I could try again with a
more granular period, but I feel like this method is ultimately a goose
chase.
Bizarre. There is no good explanation why memory usage would go
up to 32 GB (?) within one top interval (3 sec ?). My theory
about one gigantic object is debunked: you have only the 118 MB
one. Perhaps there's some container or process memory limit, as
Luke guessed, but it's not obvious here.
The other big hammer is "strace". If you're still interested in
playing with this, you could do:
strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
and hours later, see if something suggests itself toward the
end of that output file.
-- Pete
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html