Each of the two bricks has just one XFS file system w/ inode64 enabled on a 10TB LVM LV. Each of the volumes is less than 40% full and inode counts look reasonable. I'm working to get a test environment going so I can reproduce this off-production and add the trace translator. I've sent in a bunch of trace data and opened a bug: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1040 -Brian -- Brian Smith Senior Systems Administrator IT Research Computing, University of South Florida 4202 E. Fowler Ave. ENB204 Office Phone: +1 813 974-1467 Organization URL: http://rc.usf.edu > Date: Wed, 30 Jun 2010 15:33:24 -0400 > From: Brian Smith <brs at usf.edu> > Subject: Re: Revisit: FORTRAN Codes and File I/O > To: Harshavardhana <harsha at gluster.com> > Cc: Gluster General Discussion List <gluster-users at gluster.org> > Message-ID: <1277926404.2687.60.camel at localhost.localdomain> > Content-Type: text/plain; charset="UTF-8" > > Spoke too soon. Same problem occurs minus all performance translators. > Debug logs on the server show > > [2010-06-30 15:30:54] D [server-protocol.c:2104:server_create_cbk] > server-tcp: create(/b/brs/Si/CHGCAR) inode (ptr=0x2aaab00e05b0, > ino=2159011921, gen=5488651098262601749) found conflict > (ptr=0x2aaab40cca00, ino=2159011921, gen=5488651098262601749) > [2010-06-30 15:30:54] D [server-resolve.c:386:resolve_entry_simple] > server-tcp: inode (pointer: 0x2aaab40cca00 ino:2159011921) found for > path (/b/brs/Si/CHGCAR) while type is RESOLVE_NOT > [2010-06-30 15:30:54] D [server-protocol.c:2132:server_create_cbk] > server-tcp: 72: CREATE (null) (0) ==> -1 (File exists) > > -Brian > > -- > Brian Smith > Senior Systems Administrator > IT Research Computing, University of South Florida > 4202 E. Fowler Ave. ENB204 > Office Phone: +1 813 974-1467 > Organization URL: http://rc.usf.edu > > > On Wed, 2010-06-30 at 13:06 -0400, Brian Smith wrote: > > I received these in my debug output during a run that failed: > > > > [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: > > unexpected offset (8192 != 1062) resetting > > [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: > > unexpected offset (8192 != 1062) resetting > > [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: > > unexpected offset (8192 != 1062) resetting > > [2010-06-30 12:34:25] D [read-ahead.c:468:ra_readv] readahead: > > unexpected offset (8192 != 1062) resetting > > > > I disabled the read-ahead translator as well as the three other > > performance translators commented out in my vol file (I'm on GigE; the > > docs say I can still reach link max anyway) and my processes appear to > > be running smoothly. I'll go ahead and submit the bug report with > > tracing enabled as well. > > > > -Brian > > > > > Date: Wed, 30 Jun 2010 21:17:59 -0400 > From: Jeff Darcy <jdarcy at redhat.com> > Subject: Re: Revisit: FORTRAN Codes and File I/O > To: gluster-users at gluster.org > Message-ID: <4C2BECC7.5010402 at redhat.com> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > On 06/30/2010 03:33 PM, Brian Smith wrote: > > Spoke too soon. Same problem occurs minus all performance translators. > > Debug logs on the server show > > > > [2010-06-30 15:30:54] D [server-protocol.c:2104:server_create_cbk] > > server-tcp: create(/b/brs/Si/CHGCAR) inode (ptr=0x2aaab00e05b0, > > ino=2159011921, gen=5488651098262601749) found conflict > > (ptr=0x2aaab40cca00, ino=2159011921, gen=5488651098262601749) > > [2010-06-30 15:30:54] D [server-resolve.c:386:resolve_entry_simple] > > server-tcp: inode (pointer: 0x2aaab40cca00 ino:2159011921) found for > > path (/b/brs/Si/CHGCAR) while type is RESOLVE_NOT > > [2010-06-30 15:30:54] D [server-protocol.c:2132:server_create_cbk] > > server-tcp: 72: CREATE (null) (0) ==> -1 (File exists) > > > The first line almost looks like a create attempt for a file that > already exists at the server. The second and third lines look like *yet > another* create attempt, failing this time before the request is even > passed to the next translator. This might be a good time to drag out > the debug/trace translator, and sit it on top of brick1 to watch the > create calls. That will help nail down the exact sequence of events as > the server sees them, so we don't go looking in the wrong places. It > might even be useful to do the same on the client side, but perhaps not > yet. Instructions are here: > > http://www.gluster.com/community/documentation/index.php/Translators/debug/trace > > In the mean time, to further identity which code paths are most likely > to be relevant, it would be helpful to know a couple more things. > > (1) Is each storage/posix volume using just one local filesystem, or is > it possible that the underlying directory tree spans more than one? > This could lead to inode-number duplication, which requires extra handling. > > (2) Are either of the server-side volumes close to being full? This > could result in creating an extra "linkfile" on the subvolume/server > where we'd normally create the file, pointing to where we really created > it due to space considerations. > > ------------------------------ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > End of Gluster-users Digest, Vol 27, Issue 1 > ********************************************
