Hi Susant,
The cause of the issue seems to be that gluster on freebsd is unable to
correctly determine the available disk space; it is reporting that I
have 12.6PB of total disk space, when in reality I have only 40T. As
such, the bricks on that server have much more weight than those on the
other server. As a temporary workaround, I disabled
cluster.weighted-rebalance.
Output from CentOS (192.168.110.2):
[kyle@colossus ~]$ sudo gluster volume status ftp detail
Status of volume: ftp
------------------------------------------------------------------------------
Brick : Brick 192.168.110.1:/tank/bricks/ftp
TCP Port : 49159
RDMA Port : 0
Online : Y
Pid : 69547
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 2.5PB
Total Disk Space : 12.6PB
Inode Count : 21097228996
Free Inodes : 21087359717
------------------------------------------------------------------------------
Brick : Brick 192.168.110.2:/ftp/bricks/ftp
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 1087
File System : zfs
Device : storage/bricks
Mount Options : rw,noatime
Inode Size : N/A
Disk Space Free : 39.3TB
Total Disk Space : 46.3TB
Inode Count : 84298675428
Free Inodes : 84297711138
[kyle@colossus ~]$ sudo zfs list storage/bricks
NAME USED AVAIL REFER MOUNTPOINT
storage/bricks 7.05T 39.3T 7.05T /ftp/bricks
Output from FreeBSD (192.168.110.1):
$ zfs list -r tank
NAME USED AVAIL REFER MOUNTPOINT
tank 40.5T 9.82T 153K /tank
tank/bricks 40.5T 9.82T 40.5T /tank/bricks
As you can see, the Disk Space Free and Total Disk Space is way off. In
addition, gluster is not able to determine the file system, device, and
mount options.
At least regarding the device and mount options, it looks like
cli-command-volume.c only checks linux hosts (see
https://github.com/gluster/glusterfs/blob/70fd68d94f768c098b3178c151fa92c5079a8cfd/cli/src/cli-cmd-volume.c#L1980
and
https://github.com/gluster/glusterfs/blob/70fd68d94f768c098b3178c151fa92c5079a8cfd/cli/src/cli-cmd-volume.c#L1996)
I'll follow up to this thread as I find more information.
Kyle
On 07/06/2016 11:43 PM, Susant Palai wrote:
Hi,
Please pass on the rebalance log from the 1st server for more analysis which can be found under /var/log/glusterfs/"$VOL-rebalance.log".
And also we need the current layout xattrs from both the bricks, which can be extracted by the following command.
"getfattr -m . -de hex <$BRICK_PATH>".
Thanks,
Susant
----- Original Message -----
From: "Kyle Johnson" <kjohnson@xxxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Sent: Tuesday, 5 July, 2016 10:58:09 PM
Subject: Problem rebalancing a distributed volume
Hello everyone,
I am having trouble with a distributed volume. In short, the rebalance
command does not seem to work for me: Existing files are not migrated, and
new files are not created on the new brick.
I am running glusterfs 3.7.6 on two servers:
1) FreeBSD 10.3-RELEASE (colossus2 - 192.168.110.1)
2) CentOS 6.7 (colossus - 192.168.110.2)
The bricks are zfs-backed on both servers, and the network consists of two
direct-connected cat6 cables on 10gig NICs. The NICs are bonded (lagg'd)
together with mode 4 (LACP).
Here is what I am seeing:
root@colossus ~]# gluster volume create fubar 192.168.110.2:/ftp/bricks/fubar
volume create: fubar: success: please start the volume to access data
[root@colossus ~]# gluster volume start fubar
volume start: fubar: success
[root@colossus ~]# mount -t glusterfs 192.168.110.2:/fubar /mnt/test
[root@colossus ~]# touch /mnt/test/file{1..100}
[root@colossus ~]# ls / mnt/test / | wc -l
100
[root@colossus ~]# ls /ftp/bricks/fubar | wc -l
100
# So far, so good.
[root@colossus ~]# gluster volume add-brick fubar
192.168.110.1:/tank/bricks/fubar
volume add-brick: success
# For good measure, I'll do an explicit fix-layout first.
[root@colossus ~]# gluster volume rebalance fubar fix-layout start
volume rebalance: fubar: success: Rebalance on fubar has been started
successfully. Use rebalance status command to check status of the rebalance
process.
ID: 2da23238-dbe4-4759-97b2-08879db271e7
[root@colossus ~]# gluster volume rebalance fubar status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- -----------
------------ --------------
localhost 0 0Bytes 0 0 0 fix-layout completed 0.00
192.168.110.1 0 0Bytes 0 0 0 fix-layout completed 0.00
volume rebalance: fubar: success
# Now to do the actual rebalance.
[root@colossus ~]# gluster volume rebalance fubar start
volume rebalance: fubar: success: Rebalance on fubar has been started
successfully. Use rebalance status command to check status of the rebalance
process.
ID: 67160a67-01b2-4a51-9a11-114aa6269ee9
[root@colossus ~]# gluster volume rebalance fubar status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- -----------
------------ --------------
localhost 0 0Bytes 100 0 0 completed 0.00
192.168.110.1 0 0Bytes 0 0 0 completed 0.00
volume rebalance: fubar: success
[root@colossus ~]# ls / mnt/test / | wc -l
101
[root@colossus ~]# ls / ftp/bricks/fubar / | wc -l
100
# As the output shows, 100 files were scanned, but none were moved.
# And for another test, I'll create 100 new post-fix-layout files
[root@colossus ~]# touch /mnt/test/file{101..200}
[root@colossus ~]# ls / ftp/bricks/fubar / | wc -l
199
# And as you can see here, they were all created on the first server. The
second server isn't touched at all.
Not sure if this is relevant, but if I create the volume with both bricks to
begin with, files are properly distributed.
Thanks!
Kyle
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
