Please try with io-threads on the server side (between protocol/server and posix) avati 2007/6/27, Shai DB <dbshai@xxxxxxxxx>:
after copying some few thousands files and deleting and copying again i get a lot of errors: File descriptor in bad state No such file or directory and a lot of [Jun 26 05:45:13] [CRITICAL/tcp.c:81/tcp_disconnect()] transport/tcp:server: connection to server disconnected [Jun 26 05:45:13] [ERROR/common-utils.c:55/full_rw()] libglusterfs:full_rw: 0 bytes r/w instead of 113 (errno=9) in the glusterd.log I have set it up like this: 1.3-pre4 5 servers + 5 clients (running on same boxes as servers). what could cause the disconnection ? server: volume gfs type storage/posix option directory /mnt/gluster/gfs1 end-volume volume gfs-afr type storage/posix option directory /mnt/gluster/afr-gfs1 end-volume volume server type protocol/server option transport-type tcp/server option listen-port 6996 subvolumes gfs gfs-afr option auth.ip.gfs.allow * option auth.ip.gfs-afr.allow * end-volume client: volume gfs type storage/posix option directory /mnt/gluster/gfs1 end-volume volume gfs-afr type storage/posix option directory /mnt/gluster/afr-gfs1 end-volume volume server type protocol/server option transport-type tcp/server option listen-port 6996 subvolumes gfs gfs-afr option auth.ip.gfs.allow * option auth.ip.gfs-afr.allow * end-volume [root@hd-t1157cl etc]# cat cluster-client.vol volume a1 type protocol/client option transport-type tcp/client option remote-host 10.47.0.10 option remote-port 6996 option remote-subvolume gfs end-volume volume a2 type protocol/client option transport-type tcp/client option remote-host 10.47.0.10 option remote-port 6996 option remote-subvolume gfs-afr end-volume volume b1 type protocol/client option transport-type tcp/client option remote-host 10.47.0.11 option remote-port 6996 option remote-subvolume gfs end-volume volume b2 type protocol/client option transport-type tcp/client option remote-host 10.47.0.11 option remote-port 6996 option remote-subvolume gfs-afr end-volume volume c1 type protocol/client option transport-type tcp/client option remote-host 10.47.0.12 option remote-port 6996 option remote-subvolume gfs end-volume volume c2 type protocol/client option transport-type tcp/client option remote-host 10.47.0.12 option remote-port 6996 option remote-subvolume gfs-afr end-volume volume d1 type protocol/client option transport-type tcp/client option remote-host 10.47.0.13 option remote-port 6996 option remote-subvolume gfs end-volume volume d2 type protocol/client option transport-type tcp/client option remote-host 10.47.0.13 option remote-port 6996 option remote-subvolume gfs-afr end-volume volume e1 type protocol/client option transport-type tcp/client option remote-host 10.47.0.14 option remote-port 6996 option remote-subvolume gfs end-volume volume e2 type protocol/client option transport-type tcp/client option remote-host 10.47.0.14 option remote-port 6996 option remote-subvolume gfs-afr end-volume volume afr1 type cluster/afr subvolumes a1 e2 option replicate *:2 end-volume volume afr2 type cluster/afr subvolumes b1 d2 option replicate *:2 end-volume volume afr3 type cluster/afr subvolumes c1 a2 option replicate *:2 end-volume volume afr4 type cluster/afr subvolumes d1 b2 option replicate *:2 end-volume volume afr5 type cluster/afr subvolumes e1 c2 option replicate *:2 end-volume volume gfstest type cluster/unify subvolumes afr1 afr2 afr3 afr4 afr5 option scheduler rr option rr.limits.min-free-disk 5GB end-volume On 6/26/07, Sebastien LELIEVRE <slelievre@xxxxxxxxxxxxxxxx> wrote: > > Hi again ! > > Shai DB a écrit : > > another question > > I notice that 1.2 dont have the AFR on its source > > how can i use/install it anyway ? > > i saw 1.3-pre has it.. > > is the 1.3-pre OK for production ? > > thanks > > > > I had forgotten this point ! :) > > Yes, 1.3-pre4 archive is stable enough for production, but you can also > use the tla repository with the branch 2.4 which is stable enough (to > me) to be used in production. > > Just note that the 1.3 stable release will be based on the 2.5 > mainbranch and will include self-heal feature (and many more !) > > Cheers, > > Sebastien LELIEVRE > slelievre@xxxxxxxxxxxxxxxx Services to ISP > TBS-internet http://www.TBS-internet.com > > > I need it for replication (to have 2 copies of data in case of crash) > > > > > > On 6/26/07, *Sebastien LELIEVRE* <slelievre@xxxxxxxxxxxxxxxx > > <mailto:slelievre@xxxxxxxxxxxxxxxx>> wrote: > > > > Hi, > > > > I just wanted to stress this : > > > > Shai a écrit : > > > Hello, we are testing glusterfs 1.2 and I have few questions - > > > > 1.2 doesn't bring "self-heal" with it, so keep in mind that if a > drives > > crashes, you would have to sync your new drive "manually" with the > > others. > > > > > > so to just copy all data to the replaced disk from his afr 'pair' ? > > > > > > BUT, 1.3 is going to correct this, and this is good :) > > > > That's all I had to add > > > > Cheers, > > > > Sebastien LELIEVRE > > slelievre@xxxxxxxxxxxxxxxx > > <mailto:slelievre@xxxxxxxxxxxxxxxx> Services to ISP > > TBS-internet http://www.TBS-internet.com > > > > Krishna Srinivas a écrit : > > > As of now you need to restart glusterfs if there is any change > > > in the config spec file. However in future versions you wont need > > > to remount (This is in our road map) > > > > > > On 6/25/07, Shai DB <dbshai@xxxxxxxxx <mailto:dbshai@xxxxxxxxx>> > > wrote: > > >> thanks for the answer > > >> this seems easy and neat to setup > > >> > > >> another question is, if i add 2 more nodes to the gang > > >> how can i setup all the clients with the new configuration, > without > > >> need to > > >> 'remount' the glusterfs ? > > >> > > >> Thanks > > >> > > >> > > >> On 6/25/07, Krishna Srinivas <krishna@xxxxxxxxxxxxx > > <mailto:krishna@xxxxxxxxxxxxx>> wrote: > > >> > > > >> > On 6/25/07, Shai DB < dbshai@xxxxxxxxx > > <mailto:dbshai@xxxxxxxxx>> wrote: > > >> > > Hello, we are testing glusterfs 1.2 and I have few questions > - > > >> > > > > >> > > > > >> > > 1. we are going to store millions of small jpg files that > > will be > > >> read > > >> > by > > >> > > webserver - is glusterfs good solution for this ? > > >> > > > >> > Yes, definitely. > > >> > > > >> > > 2. we are going to run both server+clients on each node > > together with > > >> > apache > > >> > > > > >> > > 3. replicate *:2 > > >> > > > > >> > > the way i think doing replicate is defining on each server 2 > > >> volumes and > > >> > > using AFR: > > >> > > > > >> > > server1: a1, a2 > > >> > > server2: b1, b2 > > >> > > server3: c1, c2 > > >> > > server4: d1, d2 > > >> > > server5: e1, e2 > > >> > > > > >> > > afr1: a1+b2 > > >> > > afr2: b1+c2 > > >> > > afr3: c1+d2 > > >> > > afr4: d1+e2 > > >> > > afr5: e1+a2 > > >> > > > > >> > > and then unify = afr1+afr2+afr3+afr4+afr5 with replicate > option > > >> > > > > >> > > is this correct way ? > > >> > > and what to do on the future when we add more nodes ? when > > >> changing the > > >> > afr > > >> > > (adding and changing the couples) making glusterfs > > >> > > redistribute the files the new way ? > > >> > > > >> > Yes this is the right way. If you add one more server f, the > one > > >> solution > > >> > is to move contents of a2 to f2 and clean up a2 and have it as > > >> following: > > >> > > > >> > afr5: e1 + f2 > > >> > afr6: f1 + a2 > > >> > > > >> > Cant think of an easier solution. > > >> > > > >> > But if we assume that you will always add 2 servers when you > > want to > > >> add, > > >> > we can have the setup in following way: > > >> > afr1: a1 + b2 > > >> > afr2: b1 + a2 > > >> > afr3: c1 + d2 > > >> > afr4: d1 + c2 > > >> > afr5: e1 + f2 > > >> > afr6: f1 + e2 > > >> > > > >> > Now when you add a pair of servers to this (g, h): > > >> > afr7: f1 + h2 > > >> > afr8: h1 +f2 > > >> > > > >> > Which is very easy. But you will have to add 2 servers > everytime. > > >> > The advantage is that it is easier to visualize the setup and > add > > >> > new nodes. > > >> > > > >> > Thinking further, if we assume that you will replicate all the > > files > > >> > twice (option replicate *:2) you can have the following setup: > > >> > afr1: a + b > > >> > afr2: c + d > > >> > afr3: e + f > > >> > > > >> > This is a very easy setup. It is simple to add a fresh pair > > (afr4: g > > >> +h) > > >> > > > >> > You can have whatever setup you want depending on your > > >> > convinience and requirement. > > >> > > > >> > > > > >> > > 4. what happens when a hard drive goes down and replaces, the > > cluster > > >> > also > > >> > > redistribute the files ? > > >> > > > >> > When a hard drive is replaced, missing files will be replicated > > from > > >> the > > >> > AFR's other child. > > >> > > > >> > Regards > > >> > Krishna > > >> > > > >> > ------- > > >> > > > >> > The best quote ever : ' > > >> > > > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxx http://lists.nongnu.org/mailman/listinfo/gluster-devel
-- Anand V. Avati
