resending to list. On Nov 29, 2007 9:27 AM, Raghavendra G <raghavendra@xxxxxxxxxxxxx> wrote: > Hi Geoff, > Can you please check with glusterfs--mainline--2.5--patch-584 that the bug > still exists? > > regards, > > > On Nov 12, 2007 6:49 PM, Geoff Kassel < gkassel@xxxxxxxxxxxxxxxxxxxxx> > wrote: > > > Hi Raghavendra, > > Sorry about the delay - I've now been able to reproduce the error on > > my > > virtual server setup, where's there's no noise from production use. > > > > Here's the glusterfs.log entries from the actions in the original email: > > > > 2007-11-13 00:38:56 E [afr.c:5154:afr_closedir] shared-namespace-afr: > > afrfdp > > is NULL, returning EBADFD > > 2007-11-13 00:38:56 E [afr.c:5154:afr_closedir] shared-dataspace-afr: > > afrfdp > > is NULL, returning EBADFD > > 2007-11-13 00:39:04 E [afr.c:4412:afr_create_cbk] shared-namespace-afr: > > (path=/test/testfile child=shared-A-namespace-iot) op_ret=-1 op_errno=13 > > > > 2007-11-13 00:39:04 E [afr.c:4412:afr_create_cbk] shared-namespace-afr: > > (path=/test/testfile child=shared-B-namespace-iot) op_ret=-1 op_errno=13 > > > > This is with the second-to-latest TLA (552) and fuse-2.7.0-glfs6 . > > > > Please advise if there's any tests or particular client-server > > configurations > > you'd like me to try. > > > > Kind regards, > > > > Geoff Kassel. > > > > On Tue, 6 Nov 2007, you wrote: > > > oh.. sorry for that.. I didnt observe that. > > > > > > On Nov 6, 2007 10:04 AM, Geoff Kassel <gkassel@xxxxxxxxxxxxxxxxxxxxx > > > wrote: > > > > Please don't forward the message with my work address onto the devel > > list > > > > - > > > > this was accidental. I'd prefer to keep my work address off the > > list. > > > > > > > > On Tue, 6 Nov 2007, Geoff Kassel wrote: > > > > > To rule out an interaction with libnss-mysql, I have also tested > > this > > > > > > > > with > > > > > > > > > a passwd/groups setup on both machines, by the way. > > > > > > > > > > On Tue, 6 Nov 2007, Geoff Kassel wrote: > > > > > > Both alpha and bravo have the same group set up in this case - > > it's > > > > > > managed by libnss-mysql, on a MySQL Cluster server. > > > > > > > > > > > > On Tue, 6 Nov 2007, you wrote: > > > > > > > Another question, > > > > > > > Does both alpha and bravo have the user group you are > > experimenting > > > > > > > with? > > > > > > > > > > > > > > regards, > > > > > > > > > > > > > > On Nov 6, 2007 9:34 AM, Geoff Kassel > > > > > > > < gkassel@xxxxxxxxxxxxxxxxxxxxx> > > > > > > > > > > wrote: > > > > > > > > Hi Raghavendra, > > > > > > > > I'll see if I can do a log run for this when my system is > > > > > > > > quieter > > > > > > > > > > > > i.e. sometime later tonight, Australian Eastern Standard > > Time. I > > > > > > > > don't think I'm > > > > > > > > going to to be able to produce any clearer logs than that > > already > > > > > > > > provided, > > > > > > > > though, since this is a multi-user production system. > > > > > > > > > > > > > > > > In the meanwhile, I have a virtual server image of this > > system > > > > > > > > pre-deployment that I'm planning on testing to see if the > > bug > > > > > > > > occurs > > > > > > > > > > > > with the > > > > > > > > 545 patch. If it does, I can provide clearer logs for you > > now. If > > > > > > > > I > > > > > > > > > > > > can't get > > > > > > > > it to recur on this system, that'll narrow down quite > > > > > > > > substantially > > > > > > > > > > > > the circumstances under which this bug occurs. > > > > > > > > > > > > > > > > I'll let you know how I go. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > Geoff Kassel. > > > > > > > > > > > > > > > > On Mon, 5 Nov 2007, you wrote: > > > > > > > > > Hi Geoff, > > > > > > > > > > > > > > > > > > The glusterfs.log you sent me indicates that you've been > > doing > > > > > > > > lots > > > > > > > > > > > > > of operations. Is it possible for you to reproduce the bug > > just > > > > > > > > > using the > > > > > > > > > > > > > > > > set > > > > > > > > > > > > > > > > > of commands you had given in your first mail? i.e, > > > > > > > > > > > > > > > > > > groupmems -a nonrootuser -g randomgroup > > > > > > > > > cd /glusterfsmountpoint > > > > > > > > > mkdir test > > > > > > > > > chown root:randomgroup test > > > > > > > > > chmod 770 test > > > > > > > > > su - nonrootuser > > > > > > > > > cd /glusterfsmountpoint/test # Succeeds > > > > > > > > > ls # Permission denied error. > > > > > > > > > echo "Hello" > testfile # Also gives a perm > > > > > > > > > > > > > > > > > > and send the logs generated? > > > > > > > > > > > > > > > > > > On Nov 5, 2007 12:57 PM, Geoff Kassel > > > > > > > > > <gkassel@xxxxxxxxxxxxxxxxxxxxx> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > Hi Raghavendra, > > > > > > > > > > > > > > > > > > > > > I found two client spec files in both alpha/ and > > bravo/. by > > > > > > > > any > > > > > > > > > > > > chance > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > you running two clients? > > > > > > > > > > > > > > > > > > > > Yes, that is correct. I'm using GlusterFS to maintain > > shared > > > > > > > > > > storage between > > > > > > > > > > two machines. Both machines run a GlusterFS server, > > providing > > > > > > > > a > > > > > > > > > > > > namespace > > > > > > > > > > > > > > > > > > and > > > > > > > > > > a dataspace, which are subsequently unified and AFR'd in > > the > > > > > > > > > > client > > > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > > > run > > > > > > > > > > on each machine. > > > > > > > > > > > > > > > > > > > > > #uname -a > > > > > > > > > > > > > > > > > > > > Linux alpha 2.6.20-hardened-r10 #1 SMP Sat Oct 20 > > 05:41:43 > > > > > > > > > > EST 2007 > > > > > > > > > > > > > > > > i686 > > > > > > > > > > > > > > > > > > Dual-Core AMD Opteron(tm) Processor 2218 AuthenticAMD > > > > > > > > GNU/Linux > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > Linux bravo 2.6.20-hardened-r10 #1 SMP Sat Oct 20 > > 05:41:43 > > > > > > > > > > EST 2007 > > > > > > > > > > > > > > > > i686 > > > > > > > > > > > > > > > > > > Dual-Core AMD Opteron(tm) Processor 2218 AuthenticAMD > > > > > > > > GNU/Linux > > > > > > > > > > > > > > The two machines are identical in hardware, and the > > software > > > > > > > > > > configuration is > > > > > > > > > > more or less identical, being different only in host > > name and > > > > > > > > > > > > > > > > symmetrical > > > > > > > > > > > > > > > > > > changes for load balancing and hosting purposes. > > GlusterFS > > > > > > > > > > > > > > > > communicates > > > > > > > > > > > > > > > > > > over > > > > > > > > > > a GigE link between the pair, utilized only by GlusterFS > > and > > > > > > > > > > > > > > > > keepalived > > > > > > > > > > > > > > > > > > for > > > > > > > > > > load-balancing and maintenance of redundant services. > > > > > > > > > > > > > > > > > > > > While the processors are 64-bit, I'm running 32-bit for > > > > > > > > software > > > > > > > > > > > > > > compatibility > > > > > > > > > > reasons - that's one difference between your kernel and > > mine. > > > > > > > > I > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > > PaX, > > > > > > > > > > which affects memory allocation and access, and have the > > > > > > > > > > grsec > > > > > > > > > > > > > > > > extensions > > > > > > > > > > > > > > > > > > compiled in (as part of the Gentoo hardened-sources) but > > not > > > > > > > > > > enabled. > > > > > > > > > > > > > > > > > > > > Attached is my kernel config. I hope this helps. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > Geoff Kassel. > > > > > > > > > > > > > > > > > > > > On Mon, 5 Nov 2007, you wrote: > > > > > > > > > > > Hi Geoff, > > > > > > > > > > > > > > > > > > > > > > I found two client spec files in both alpha/ and > > bravo/. by > > > > > > > > any > > > > > > > > > > > > chance > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > you running two clients? > > > > > > > > > > > > > > > > > > > > > > I tried with single glusterfs client running on a node > > with > > > > > > > > the > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > > #uname -a > > > > > > > > > > > Linux master-node 2.6.18-8.el5 #1 SMP Thu Mar 15 > > 19:46:53 > > > > > > > > EDT > > > > > > > > > > > > > > > 2007 > > > > > > > > > > > > > > > > > > > > x86_64 > > > > > > > > > > > > > > > > > > > > > x86_64 x86_64 GNU/Linux > > > > > > > > > > > > > > > > > > > > > > but still I cannot reproduce your problem. I am using > > > > > > > > > > > > > > > > fuse-2.7.0-glfs5. > > > > > > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > the only difference in configuration of the node is > > the > > > > > > > > kernel. > > > > > > > > > > > > > > > Going through logs If I can find any hints. > > > > > > > > > > > regards, > > > > > > > > > > > > > > > > > > > > > > On Nov 5, 2007 10:31 AM, Geoff Kassel > > > > > > > > > > > <gkassel@xxxxxxxxxxxxxxxxxxxxx > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > Hi Raghavendra, > > > > > > > > > > > > > > > > > > > > > > > > I'm not replying to the list because of the size of > > the > > > > > > > > > > > > attached > > > > > > > > > > > > > > > > log > > > > > > > > > > > > > > > > > > > > files. > > > > > > > > > > > > > > > > > > > > > > > > > I tried to reproduce your problem with > > > > > > > > > > > > > > > > > > > > > > > > glusterfs--mainline--2.5--patch-545, > > > > > > > > > > > > > > > > > > > > > > > > > but without success. > > > > > > > > > > > > > > > > > > > > > > > > That's not too surprising - I use a 2.6.20 hardened > > > > > > > > > > > > Gentoo kernel with PaX enabled, so I'm likely to get > > > > > > > > > > > > different errors to > > > > > > > > > > > > > > > > everyone > > > > > > > > > > > > > > > > > > > > else. I found the > > > > > > > > > > > > error occurred with (at least) patches 543 and > > onwards. > > > > > > > > > > > > > > > > > > > > > > > > > Can you send your client/server logs and > > configuration > > > > > > > > > > > > > files? Also what is the fuse version you are > > using? > > > > > > > > > > > > > > > > > > > > > > > > Sure - they're in the attached ZIP file. I'm using > > FUSE > > > > > > > > > > > > > > > > 2.7.0-glfs5 > > > > > > > > > > > > > > > > > > > > on both > > > > > > > > > > > > machines, known as alpha and bravo. I've sent logs > > and > > > > > > > > spec > > > > > > > > > > > > > > > > files > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > both machines, in the accordingly named directories. > > > > > > > > > > > > > > > > > > > > > > > > > > To find the time of my various tests, search for ': > > > > > > > > > > > > /test' > > > > > > > > in > > > > > > > > > > > > > > > > the glusterfs > > > > > > > > > > > > logs. The most relevant test (i.e. the results of > > which I > > > > > > > > > > > > posted > > > > > > > > > > > > > > > > > > > > about) > > > > > > > > > > > > > > > > > > > > > > is ': /test4'. > > > > > > > > > > > > > > > > > > > > > > > > Good luck with the bug hunt! > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > Geoff Kassel. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 5 Nov 2007, Raghavendra G wrote: > > > > > > > > > > > > > Hi Geoff, > > > > > > > > > > > > > I tried to reproduce your problem with > > > > > > > > > > > > > > > > > > > > > > > > glusterfs--mainline--2.5--patch-545, > > > > > > > > > > > > > > > > > > > > > > > > > but without success. Can you send your > > client/server > > > > > > > > logs > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > > > > > > > > files? Also what is the fuse version you are > > using? > > > > > > > > > > > > > > > > > > > > > > > > > > regards, > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 4, 2007 4:39 PM, Geoff Kassel > > > > > > > > > > > > > <gkassel@xxxxxxxxxxxxxxxxxxxxx> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I've think I've found a group permissions bug > > in > > > > > > > > > > > > > > the latest > > > > > > > > > > > > > > > > > > > > patches > > > > > > > > > > > > > > > > > > > > > > > > committed to the repository. The bug causes > > > > > > > > > > > > > > permission denied > > > > > > > > > > > > > > > > > > > > errors > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > non-root users where they should have adequate > > > > > > > > > > > > > > access. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The following set of commands under a glusterfs > > mount > > > > > > > > > > > > > > > > reproduces > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > bug > > > > > > > > > > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > me: > > > > > > > > > > > > > > > > > > > > > > > > > > > > groupmems -a nonrootuser -g randomgroup > > > > > > > > > > > > > > cd /glusterfsmountpoint > > > > > > > > > > > > > > mkdir test > > > > > > > > > > > > > > chown root:randomgroup test > > > > > > > > > > > > > > chmod 770 test > > > > > > > > > > > > > > su - nonrootuser > > > > > > > > > > > > > > cd /glusterfsmountpoint/test # Succeeds > > > > > > > > > > > > > > ls # Permission denied error. > > > > > > > > > > > > > > echo "Hello" > testfile # Also gives a > > > > > > > > permission > > > > > > > > > > > > denied > > > > > > > > > > > > > > > > > > > > > > error. > > > > > > > > > > > > > > > > > > > > > > > > > > > > To get a successful ls under the non-root user, > > > > > > > > > > > > > > permissions > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > be 775 in > > > > > > > > > > > > > > the test directory. To get a successful file > > > > > > > > > > > > > > creation, > > > > > > > > it > > > > > > > > > > > > needs > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > 777. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have noticed that on the initial cd as the > > non-root > > > > > > > > > > > > > > user, > > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > > > > seems > > > > > > > > > > > > > > > > > > > > > > > > > > to be > > > > > > > > > > > > > > the following in the client logs (I'm using > > > > > > > > AFR/Unify): > > > > > > > > > > > > > > 2007-11-04 22:36:25 E [afr.c:5654:afr_closedir] > > > > > > > > > > > > > > > > > > > > shared-namespace-afr: > > > > > > > > > > > > > > afrfdp > > > > > > > > > > > > > > is NULL, returning EBADFD > > > > > > > > > > > > > > 2007-11-04 22:36:25 E [ afr.c:5654:afr_closedir] > > > > > > > > > > > > > > > > > > > > shared-dataspace-afr: > > > > > > > > > > > > > > afrfdp > > > > > > > > > > > > > > is NULL, returning EBADFD > > > > > > > > > > > > > > 2007-11-04 22:36:25 E [fuse-bridge.c > > :654:fuse_fd_cbk] > > > > > > > > > > > > > > > > > > > > glusterfs-fuse: > > > > > > > > > > > > > > 1322194: /test => -1 (13) > > > > > > > > > > > > > > > > > > > > > > > > > > > > This bug doesn't cause the server or client to > > crash. > > > > > > > > (I > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > noticed > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > lot > > > > > > > > > > > > > > of 'E [ unify.c:145:unify_buf_cbk] shared: > > > > > > > > > > > > > > shared-namespace-afr > > > > > > > > > > > > > > > > > > > > > > > > returned > > > > > > > > > > > > > > > > > > > > > > > > > > 107' > > > > > > > > > > > > > > and random server crashes in general lately, but > > this > > > > > > > > > > > > > > appears > > > > > > > > > > > > > > > > > > > > > > > > unrelated.) > > > > > > > > > > > > > > > > > > > > > > > > > > I can provide logs as well as client and server > > spec > > > > > > > > > > > > > > files on request. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Geoff Kassel. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > Gluster-devel mailing list > > > > > > > > > > > > > > Gluster-devel@xxxxxxxxxx > > > > > > > > > > > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-deve > > > > > > > > > > > > > >l > > > > > > > -- > Raghavendra G > > A centipede was happy quite, until a toad in fun, > Said, "Prey, which leg comes after which?", > This raised his doubts to such a pitch, > He fell flat into the ditch, > Not knowing how to run. > -Anonymous > -- Raghavendra G A centipede was happy quite, until a toad in fun, Said, "Prey, which leg comes after which?", This raised his doubts to such a pitch, He fell flat into the ditch, Not knowing how to run. -Anonymous
