Good find! That seems to be exactly the issue. My inode numbers for the files in that directory are literally just 100k away from the 32-bit cut-off. I've rm'ed and re-extraced to get lesser inode values and voila! I guess intel must have thought that 2^32 inodes outta be enough for everyone. Many thanks, -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB308 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu On Sun, 2010-01-17 at 13:12 -0500, Aaron Knister wrote: > I ran into a similar problem when compiling code on a gluster mount > with the intel compiler. The problem was intermittent and after > digging into it further it appeared to only happy when compiling from > source files that had very high inode numbers. There's a command run > by icc that's a 32 bit executable and my guess is that somewhere > internally the extremely large inode number wasn't able to be handled > and caused the compilation to fail. The "fix" was to upgrade the > compiler. I can verify that version 11.0.064 works without a hitch (at > least in this regard). > > > More Detail: > > > I can reproduce this error with version 10.0.026 of the intel > compiler. > > > I have file test.c on a glusterfs with inode num 21479794854 (greater > than 32 bits) > 21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c > > > Running icc fails: > ... > $ icc test.c > Catastrophic error: could not open source file "test.c" > > > compilation aborted for test.c (code 4) > ... > > > However if I copy the same file to a directory on the same fs that has > a 32 bit inode number (le 4294967295) (hoping the new file will also > have a 32 bit inode number): > 11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c > > > Compilation is a success: > ... > $ !icc > icc test.c > $ echo $? > 0 > ... > > > > > If you run icc with the -v option you'll see that a program called > mcpcom gets run and produces the error. At least with version 10.0.026 > it was a 32 bit program: > $ file /opt/intel/cce/10.0.026/bin/mcpcom > /opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel > 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses > shared libs), for GNU/Linux 2.2.5, not stripped > > > Anyway I hope that helps. > > > -Aaron > > > > On Jan 17, 2010, at 11:13 AM, Brian Smith wrote: > > > Hi, > > > > I'm new to Gluster, so please forgive any ignorance on my part. > > I've > > tried reading over everything I could find to resolve this myself. > > > > I'm having an issue building executables on our gluster volume > > (there > > are some other similar issues as well, but we'll deal with this one > > first). I have the presta infiniband utilities extracted into a > > directory in my glusterfs volume and I'm attempting to build them. > > One > > of the source files, util.c (coincidentally, the first one > > referenced in > > the make file) fails to build. I get an error message: > > > > Catastrophic error: could not open source file "util.c" > > > > Some further investigating reveals the command that causes the > > issue: > > > > icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread > > -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib > > -lmpi > > -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath > > -lnuma > > -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c > > Catastrophic error: could not open source file "util.c" > > > > compilation aborted for util.c (code 4) > > > > And further, we see an strace of the command: > > ... > > 2717 6601 brk(0x957b000) = 0x957b000 > > 2718 6601 open("util.c", O_RDONLY) = 3 > > 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600, > > st_size=14370, ...}) > > = 0 > > 2720 6601 close(3) = 0 > > 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, > > 1), ...}) = 0 > > 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE| > > MAP_ANONYMOUS, -1, 0x1000) = 0xfffffffff0bce000 > > 2723 6601 write(2, "Catastrophic error: could not open source file > > \"util.c\"\n", 56) = 56 > > 2724 6601 write(2, "\n", 1) = 1 > > 2725 6601 exit_group(4) > > ... > > > > We can also see that the file is indeed present: > > > > $ ls -l util.c > > -rw------- 1 brs users 14370 Apr 8 2002 util.c > > $ stat util.c > > File: `util.c' > > Size: 14370 Blocks: 32 IO Block: 4096 regular file > > Device: 18h/24d Inode: 4295774314 Links: 1 > > Access: (0600/-rw-------) Uid: (1229806/ brs) Gid: (10001/ > > users) > > Access: 2010-01-17 00:27:43.000000000 -0500 > > Modify: 2002-04-08 21:13:57.000000000 -0400 > > Change: 2010-01-17 00:27:43.000000000 -0500 > > > > If I extract the tar-ball in /tmp, a local ext3 fs, the compile line > > above works correctly. > > > > diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c > > > > also appears clean. Any ideas what is happening here? Is there an > > issue with mmap2() and glusterfs? > > > > Many thanks in advance, > > > > -Brian > > > > > > > > -- > > Brian Smith > > Senior Systems Administrator > > IT Research Computing, University of South Florida > > 4202 E. Fowler Ave. ENB308 > > Office Phone: +1 813 974-1467 > > Organization URL: http://rc.usf.edu > > > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > >
