I ran into a similar problem when compiling code on a gluster mount with the intel compiler. The problem was intermittent and after digging into it further it appeared to only happy when compiling from source files that had very high inode numbers. There's a command run by icc that's a 32 bit executable and my guess is that somewhere internally the extremely large inode number wasn't able to be handled and caused the compilation to fail. The "fix" was to upgrade the compiler. I can verify that version 11.0.064 works without a hitch (at least in this regard).
More Detail:
I can reproduce this error with version 10.0.026 of the intel compiler.
I have file test.c on a glusterfs with inode num 21479794854 (greater than 32 bits)
21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
Running icc fails:
...
$ icc test.c
Catastrophic error: could not open source file "test.c"
compilation aborted for test.c (code 4)
...
However if I copy the same file to a directory on the same fs that has a 32 bit inode number (le 4294967295) (hoping the new file will also have a 32 bit inode number):
11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
Compilation is a success:
...
$ !icc
icc test.c
$ echo $?
0
...
If you run icc with the -v option you'll see that a program called mcpcom gets run and produces the error. At least with version 10.0.026 it was a 32 bit program:
$ file /opt/intel/cce/10.0.026/bin/mcpcom
/opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped
Anyway I hope that helps.
-Aaron
On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
> Hi,
>
> I'm new to Gluster, so please forgive any ignorance on my part. I've
> tried reading over everything I could find to resolve this myself.
>
> I'm having an issue building executables on our gluster volume (there
> are some other similar issues as well, but we'll deal with this one
> first). I have the presta infiniband utilities extracted into a
> directory in my glusterfs volume and I'm attempting to build them. One
> of the source files, util.c (coincidentally, the first one referenced in
> the make file) fails to build. I get an error message:
>
> Catastrophic error: could not open source file "util.c"
>
> Some further investigating reveals the command that causes the issue:
>
> icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
> -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
> -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
> Catastrophic error: could not open source file "util.c"
>
> compilation aborted for util.c (code 4)
>
> And further, we see an strace of the command:
> ...
> 2717 6601 brk(0x957b000) = 0x957b000
> 2718 6601 open("util.c", O_RDONLY) = 3
> 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600, st_size=14370, ...})
> = 0
> 2720 6601 close(3) = 0
> 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> 1), ...}) = 0
> 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> MAP_ANONYMOUS, -1, 0x1000) = 0xfffffffff0bce000
> 2723 6601 write(2, "Catastrophic error: could not open source file
> \"util.c\"\n", 56) = 56
> 2724 6601 write(2, "\n", 1) = 1
> 2725 6601 exit_group(4)
> ...
>
> We can also see that the file is indeed present:
>
> $ ls -l util.c
> -rw------- 1 brs users 14370 Apr 8 2002 util.c
> $ stat util.c
> File: `util.c'
> Size: 14370 Blocks: 32 IO Block: 4096 regular file
> Device: 18h/24d Inode: 4295774314 Links: 1
> Access: (0600/-rw-------) Uid: (1229806/ brs) Gid: (10001/
> users)
> Access: 2010-01-17 00:27:43.000000000 -0500
> Modify: 2002-04-08 21:13:57.000000000 -0400
> Change: 2010-01-17 00:27:43.000000000 -0500
>
> If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> above works correctly.
>
> diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
>
> also appears clean. Any ideas what is happening here? Is there an
> issue with mmap2() and glusterfs?
>
> Many thanks in advance,
>
> -Brian
>
>
>
> --
> Brian Smith
> Senior Systems Administrator
> IT Research Computing, University of South Florida
> 4202 E. Fowler Ave. ENB308
> Office Phone: +1 813 974-1467
> Organization URL: http://rc.usf.edu
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
