Mike, I know Allen had sent you something within the support system regarding this. Based on his email, it sounds to me like it is possible something underneath Gluster has caused the rebalance to hang. This isn't any sort of fatal problem as you can simply restart the rebalance once that has been resolved and it will resume what it was doing. The amount of time it takes to rebalance depends on a few things like the number of files, the size of the data, the network speed, and also the extent to which things were unbalanced(if you are adding a lot more servers as opposed to a couple it might take longer). Generally speaking, if you double the number of servers then the rebalance will probably move half of your data, so it will take as long as that would normally take. Jacob Shucart | Gluster Systems Engineer E-Mail : Jacob at gluster.com Direct : (408)770-1504 -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Michael Robbert Sent: Monday, December 06, 2010 4:51 PM To: gluster-users at gluster.org Subject: gluster rebalance taking multiple days How long should a rebalance take? I know that it depends so lets take this example. 4 servers, 1 brick per server. here is the df -i output from the servers: [root at ra5 ~]# pdsh -g rack7 "df -i|grep brick" iosrv-7-1: 366288896 2720139 363568757 1% /mnt/brick1 iosrv-7-4: 366288896 3240868 363048028 1% /mnt/brick4 iosrv-7-2: 366288896 2594165 363694731 1% /mnt/brick2 iosrv-7-3: 366288896 3267152 363021744 1% /mnt/brick3 So, it looks like there are roughly 10 million files. I have a rebalance running on one of the servers since last Friday and this is what the status looks like right now: [root at iosrv-7-2 ~]# gluster volume rebalance gluster-test status rebalance step 1: layout fix in progress: fixed layout 149531740 As a side note I started this rebalance when I noticed that about half of my clients are missing a certain set of files. Upon further investigation I found that a different set of clients are missing different data. This problem happened after many problems getting an upgrade to 3.1.1 working. Unfortunately I don't remember which version was running when I was last able to write to this volume. Any thoughts? Mike _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
