Christopher Jefferson wrote:
On 27 May 2009, at 15:01, Tomas Carnecky wrote:
The problem appears to be the different maximum mmap sizes available
on different OSes. Whic I don't really mind the maximum file size
restriction git imposes, this restriction varying from OS to OS is
very annoying, fixing this required rewriting history to remove the
commit, which caused problems as the commit had already been pulled,
and built on, by a number of developers.
If the requirement that all files can be mmapped cannot be easily
removed, would be it perhaps be acceptable to impose a (soft?)
1GB(ish) file size limit? I suggest 1GB as all the OSes I can get
hold of easily (freeBSD, windows, Mac OS X, linux) support a mmap of
size > 1GB.
I think this is a limitation of a 32bit build of git. I just tried
with a 64bit build and it added the file just fine. The compiler on
MacOSX (gcc) produces 32bit builds by default, even if the system
supports 64bit executables. But gcc on 64bit Linux (at least the
installations I have at home) produces a 64bit executables by default.
Solaris/OpenSolaris behaves like MacOSX, no idea about *BSD or
Windows. Maybe this is why git works on Linux but not MacOSX even on
the same hardware.
Btw, I built git with: make install prefix=... CC="gcc -m64", no
modifications needed (MacOSX 10.5.7).
The git installs I am using are all 32bit, this machine doesn't have a
64bit processor (it is one of the few macs released without one). It's
nice to know long term this problem will go away, that all suggests
introducing some limit is not approriate, as while 32bit users have some
arbitary limit above which they cannot go, I am sure all 64-bit OSes
will manage to easily mmap any file. Of course warning such users they
are producing packs that are not going to work on 32bit compiles of git
isn't a stupid idea.
mmap()'ing large files (> 4GB) work just fine on Linux. You can't mmap()
more than 4GB at a time though (I think; I didn't try), but since we
don't do that anyway I doubt that was the problem.
The file you produced with your dd command should have ended up being
1239MB, or 1.21GB, so the real hard limit for MacOSX seem to be 1GB
if, indeed, there is one. On the other hand, the error message you
got ("fatal: Out of memory, malloc failed") seems to indicate the
system actually had no memory left when you tried to garbage-collect
your repository. Are you using a dual-core system? If so, please try
again with
pack.threads = 1
set in the .git/config file of that particular repository. Each
thread will allocate roughly the same amount of memory, so if
both of them had to handle that huge blob at the same time, they'd
have exploded memory usage up to 1.3GB + the compressed size of
them + DAG-bookkeeping etc etc.
I'm guessing we'd have seen error reports from other OSX users
if it was actually impossible to mmap() 1GB files in git on OSX.
--
Andreas Ericsson andreas.ericsson@xxxxxx
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
Register now for Nordic Meet on Nagios, June 3-4 in Stockholm
http://nordicmeetonnagios.op5.org/
Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html