In case it's helpful, when 3 nodes are started, the mount dir (/mnt/glusterfs) looks like: drwxrwxrwx 2 root root 49152 Feb 5 19:14 glusterfs After the 4th node it looks like: ?--------- ? ? ? ? ? /mnt/glusterfs Also, my command line is: /usr/local/sbin/glusterfs -f /usr/local/etc/glusterfs/cloud-no-repl.2.vol /mnt/glusterfs ________________________________ From: Krishna Srinivas <krishna at zresearch.com> To: Craig Flockhart <craigflockhart at yahoo.com> Cc: Amar Tumballi (bulde) <amar at gluster.com>; gluster-users at gluster.org Sent: Saturday, February 7, 2009 3:14:54 AM Subject: Re: starting 4th node in 4 node dht cluster fails Craig, Delete the backend directories (or remove trusted.glusterfs.dht xattr on them and empty the backend directories) and are create them and then start DHT and see if it works fine. Krishna On Sat, Feb 7, 2009 at 4:41 AM, Craig Flockhart <craigflockhart at yahoo.com> wrote: > Hi Amar, > Thanks for the quick reply, but that doesn't work either. I just get more > holes and overlaps: > > 2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found > anomalies in /. holes=3 overlaps=9 > 2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing > assignment on / > 2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the > directory is not a virgin > 2009-02-06 15:04:04 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: > revalidate of / failed (Structure needs cleaning) > 2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found > anomalies in /. holes=3 overlaps=9 > 2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing > assignment on / > 2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the > directory is not a virgin > 2009-02-06 15:04:04 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 2: > LOOKUP() / => -1 (Structure needs cleaning) > 2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found > anomalies in /. holes=3 overlaps=9 > 2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing > assignment on / > 2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the > directory is not a virgin > 2009-02-06 15:04:04 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: > revalidate of / failed (Structure needs cleaning) > 2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found > anomalies in /. holes=3 overlaps=9 > 2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing > assignment on / > 2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the > directory is not a virgin > 2009-02-06 15:04:04 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 3: > LOOKUP() / => -1 (Structure needs cleaning) > ________________________________ > From: Amar Tumballi (bulde) <amar at gluster.com> > To: Craig Flockhart <craigflockhart at yahoo.com> > Cc: gluster-users at gluster.org > Sent: Friday, February 6, 2009 2:37:08 PM > Subject: Re: starting 4th node in 4 node dht cluster fails > > Hi Craig, > As you are using 'distribute' (client side) over 'distribute' (server > side), this will not be working right now. To get it working right now, you > can have 4 export volumes from each server exported, and in client have 4x4 > client protocol volumes, which you can aggregate with a single > 'cluster/distribute' (which will have 16 subvolumes). > > To get the below mentioned configuration working as is, you need to wait for > a week more IMO. > > Regards, > Amar > > 2009/2/6 Craig Flockhart <craigflockhart at yahoo.com> >> >> Using dht translator to cluster together 4 nodes each with 4 disks. >> Starting glusterfs on the 4th causes "Structure needs cleaning" when >> ls-ing the mount point on any of them. It's fine with 3 nodes started. >> Using fuse-2.7.4 >> GlusterFS 2.0.0rc1 >> Linux 2.6.18-53.el5 kernel >> >> Errors from the log: >> >> >> 2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found >> anomalies in /. holes=1 overlaps=3 >> 2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing >> assignment on / >> 2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: >> the directory is not a virgin >> 2009-02-06 15:23:51 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: >> revalidate of / failed (Structure needs cleaning) >> 2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found >> anomalies in /. holes=1 overlaps=3 >> 2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing >> assignment on / >> 2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: >> the directory is not a virgin >> 2009-02-06 15:23:51 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: >> 2: LOOKUP() / => -1 (Structure needs cleaning) >> >> Config for one of the machines: >> >> volume posix-d1 >> type storage/posix >> option directory /mnt/chard1/export >> end-volume >> >> volume locks1 >> type features/locks >> subvolumes posix-d1 >> end-volume >> >> >> volume posix-d2 >> type storage/posix >> option directory /mnt/chard2/export >> end-volume >> >> >> volume locks2 >> type features/locks >> subvolumes posix-d2 >> end-volume >> >> >> volume posix-d3 >> type storage/posix >> option directory /mnt/chard3/export >> end-volume >> >> volume locks3 >> type features/locks >> subvolumes posix-d3 >> end-volume >> >> >> volume posix-d4 >> type storage/posix >> option directory /mnt/chard4/export >> end-volume >> >> volume locks4 >> type features/locks >> subvolumes posix-d4 >> end-volume >> >> volume home-ns >> type storage/posix >> option directory /var/local/glusterfs/namespace1 >> end-volume >> >> volume home >> type cluster/distribute >> subvolumes locks1 locks2 locks3 locks4 >> end-volume >> >> volume server >> type protocol/server >> option transport-type tcp >> subvolumes home >> option auth.addr.home.allow * >> end-volume >> >> >> volume zwei >> type protocol/client >> option transport-type tcp >> option remote-host zwei >> option remote-subvolume home >> end-volume >> >> volume char >> type protocol/client >> option transport-type tcp >> option remote-host char >> option remote-subvolume home >> end-volume >> >> volume pente >> type protocol/client >> option transport-type tcp >> option remote-host pente >> option remote-subvolume home >> end-volume >> >> volume tres >> type protocol/client >> option transport-type tcp >> option remote-host tres >> option remote-subvolume home >> end-volume >> >> volume dist1 >> type cluster/distribute >> subvolumes pente char tres zwei >> end-volume >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >> > > > > -- > Amar Tumballi > Gluster/GlusterFS Hacker > [bulde on #gluster/irc.gnu.org] > http://www.zresearch.com - Commoditizing Super Storage! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zresearch.com/pipermail/gluster-users/attachments/20090209/2bb570c8/attachment.htm
