open() issues on 3.0.0

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Good find!  That seems to be exactly the issue.  My inode numbers for
the files in that directory are literally just 100k away from the 32-bit
cut-off.  I've rm'ed and re-extraced to get lesser inode values and
voila!  I guess intel must have thought that 2^32 inodes outta be enough
for everyone.

Many thanks,
-Brian

-- 
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu


On Sun, 2010-01-17 at 13:12 -0500, Aaron Knister wrote:
> I ran into a similar problem when compiling code on a gluster mount
> with the intel compiler. The problem was intermittent and after
> digging into it further it appeared to only happy when compiling from
> source files that had very high inode numbers. There's a command run
> by icc that's a 32 bit executable and my guess is that somewhere
> internally the extremely large inode number wasn't able to be handled
> and caused the compilation to fail. The "fix" was to upgrade the
> compiler. I can verify that version 11.0.064 works without a hitch (at
> least in this regard).
> 
> 
> More Detail:
> 
> 
> I can reproduce this error with version 10.0.026 of the intel
> compiler.
> 
> 
> I have file test.c on a glusterfs with inode num 21479794854 (greater
> than 32 bits)
> 21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
> 
> 
> Running icc fails:
> ...
> $ icc test.c 
> Catastrophic error: could not open source file "test.c"
> 
> 
> compilation aborted for test.c (code 4)
> ...
> 
> 
> However if I copy the same file to a directory on the same fs that has
> a 32 bit inode number (le 4294967295) (hoping the new file will also
> have a 32 bit inode number):
> 11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
> 
> 
> Compilation is a success:
> ...
> $ !icc
> icc test.c 
> $ echo $?
> 0
> ...
> 
> 
> 
> 
> If you run icc with the -v option you'll see that a program called
> mcpcom gets run and produces the error. At least with version 10.0.026
> it was a 32 bit program:
> $ file /opt/intel/cce/10.0.026/bin/mcpcom
> /opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel
> 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses
> shared libs), for GNU/Linux 2.2.5, not stripped
> 
> 
> Anyway I hope that helps.
> 
> 
> -Aaron
> 
> 
> 
> On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
> 
> > Hi,
> > 
> > I'm new to Gluster, so please forgive any ignorance on my part.
> >  I've
> > tried reading over everything I could find to resolve this myself.
> > 
> > I'm having an issue building executables on our gluster volume
> > (there
> > are some other similar issues as well, but we'll deal with this one
> > first).  I have the presta infiniband utilities extracted into a
> > directory in my glusterfs volume and I'm attempting to build them.
> >  One
> > of the source files, util.c (coincidentally, the first one
> > referenced in
> > the make file) fails to build.  I get an error message:
> > 
> > Catastrophic error: could not open source file "util.c"
> > 
> > Some further investigating reveals the command that causes the
> > issue:
> > 
> > icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> > -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib
> > -lmpi
> > -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath
> > -lnuma
> > -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c 
> > Catastrophic error: could not open source file "util.c"
> > 
> > compilation aborted for util.c (code 4)
> > 
> > And further, we see an strace of the command:
> > ...
> > 2717 6601  brk(0x957b000)                    = 0x957b000
> > 2718 6601  open("util.c", O_RDONLY)          = 3
> > 2719 6601  stat64("util.c", {st_mode=S_IFREG|0600,
> > st_size=14370, ...})
> > = 0
> > 2720 6601  close(3)                          = 0
> > 2721 6601  fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> > 1), ...}) = 0
> > 2722 6601  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> > MAP_ANONYMOUS, -1, 0x1000) = 0xfffffffff0bce000
> > 2723 6601  write(2, "Catastrophic error: could not open source file
> > \"util.c\"\n", 56) = 56
> > 2724 6601  write(2, "\n", 1)                 = 1
> > 2725 6601  exit_group(4)
> > ...
> > 
> > We can also see that the file is indeed present:
> > 
> > $ ls -l util.c
> > -rw------- 1 brs users 14370 Apr  8  2002 util.c
> > $ stat util.c
> >  File: `util.c'
> >  Size: 14370      Blocks: 32         IO Block: 4096   regular file
> > Device: 18h/24d Inode: 4295774314  Links: 1
> > Access: (0600/-rw-------)  Uid: (1229806/     brs)   Gid: (10001/
> > users)
> > Access: 2010-01-17 00:27:43.000000000 -0500
> > Modify: 2002-04-08 21:13:57.000000000 -0400
> > Change: 2010-01-17 00:27:43.000000000 -0500
> > 
> > If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> > above works correctly.
> > 
> > diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
> > 
> > also appears clean.  Any ideas what is happening here?  Is there an
> > issue with mmap2() and glusterfs?
> > 
> > Many thanks in advance,
> > 
> > -Brian
> > 
> > 
> > 
> > -- 
> > Brian Smith
> > Senior Systems Administrator
> > IT Research Computing, University of South Florida
> > 4202 E. Fowler Ave. ENB308
> > Office Phone: +1 813 974-1467
> > Organization URL: http://rc.usf.edu
> > 
> > 
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > 
> 



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux