nufa helps you write to local brick, if replication is involved it will still copy it to other bricks (or suppose to do so)
what might be happening is that when initial file was created other nodes were down and it didn't replicate properly and now heal is failing
check your
gluster vol heal Volname info
maybe you will find out where second copy of the file suppose to be - and just copy it to that brick
On Sun, Oct 28, 2018 at 6:07 PM Ingo Fischer <nl@xxxxxxxxxxxxx> wrote:
Hi All,
has noone an idea on system.affinity/distributed.migrate-data ?
Or how to correctly enable nufa?
BTW: the used gluster version is 4.1.5
Thank you for your help on this!
Ingo
Am 24.10.18 um 12:54 schrieb Ingo Fischer:
> Hi,
>
> I have setup a glusterfs volume gv0 as distributed/replicated:
>
> root@pm1:~# gluster volume info gv0
>
> Volume Name: gv0
> Type: Distributed-Replicate
> Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x 2 = 6
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.178.50:/gluster/brick1/gv0
> Brick2: 192.168.178.76:/gluster/brick1/gv0
> Brick3: 192.168.178.50:/gluster/brick2/gv0
> Brick4: 192.168.178.81:/gluster/brick1/gv0
> Brick5: 192.168.178.50:/gluster/brick3/gv0
> Brick6: 192.168.178.82:/gluster/brick1/gv0
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
>
>
> root@pm1:~# gluster volume status
> Status of volume: gv0
> Gluster process TCP Port RDMA Port Online Pid
> ------------------------------------------------------------------------------
> Brick 192.168.178.50:/gluster/brick1/gv0 49152 0 Y
> 1665
> Brick 192.168.178.76:/gluster/brick1/gv0 49152 0 Y
> 26343
> Brick 192.168.178.50:/gluster/brick2/gv0 49153 0 Y
> 1666
> Brick 192.168.178.81:/gluster/brick1/gv0 49152 0 Y
> 1161
> Brick 192.168.178.50:/gluster/brick3/gv0 49154 0 Y
> 1679
> Brick 192.168.178.82:/gluster/brick1/gv0 49152 0 Y
> 1334
> Self-heal Daemon on localhost N/A N/A Y
> 5022
> Self-heal Daemon on 192.168.178.81 N/A N/A Y
> 935
> Self-heal Daemon on 192.168.178.82 N/A N/A Y
> 1057
> Self-heal Daemon on pm2.fritz.box N/A N/A Y
> 1651
>
>
> I use the fs to store VM files, so not many, but big files.
>
> The distribution now put 4 big files on one brick set and only one file
> on an other. This means that the one brick set it "overcommited" now as
> soon as all VMs using max space. SO I would like to manually
> redistribute the files a bit better.
>
> After log googling I found that the following should work:
> setfattr -n 'system.affinity' -v $location $filepath
> setfattr -n 'distribute.migrate-data' -v 'force' $filepath
>
> But I have problems with it because it gives errors or doing nothing at all.
>
> The mounting looks like:
> 192.168.178.50:gv0 on /mnt/pve/glusterfs type fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>
>
> Here is what I tried for the first xattr:
>
> root@pm1:~# setfattr -n 'system.affinity' -v 'gv0-client-5'
> /mnt/pve/glusterfs/201/imagesvm201.qcow2
> setfattr: /mnt/pve/glusterfs/201/imagesvm201.qcow2: Operation not supported
>
> So I found on google to use trusted.affinity instead and yes this works.
> I'm only not sure if the location "gv0-client-5" is correct to move the
> file to "Brick 5" from "gluster volume info gv0" ... or how this
> location is build?
> Commit Message from http://review.gluster.org/#/c/glusterfs/+/5233/ says
>> The value is the internal client or AFR brick name where you want the
> file to be.
>
> So what do I need to set there? maybe I do need the "afr" because
> replicated? But where to get that name from?
> I also tried to enter other client or replicate names like
> "gv0-replicate-0" or such which seems to be more fitting for a
> replicated volume, but result the same.
>
>
> For the second command I get:
> root@pm1:~# setfattr -n 'distribute.migrate-data' -v 'force'
> /mnt/pve/glusterfs/201/imagesvm201.qcow2
> setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: Operation
> not supported
> root@pm1:~# setfattr -n 'trusted.distribute.migrate-data' -v 'force'
> /mnt/pve/glusterfs/201/imagesvm201.qcow2
> setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: File exists
>
> I also experimented with other "names" then "gv0-client-5" above but
> always the same.
>
> I saw that instead of the second command I could start a rebalance with
> force, but this also did nothing. Ended after max1 second and moved nothing.
>
> Can someone please advice how to do it right?
>
>
> An other idea was to enable nufa and kind of "re-copy" the files on the
> glusterfs, but here it seems that the documentation is wrong.
> gluster volume set gv0 cluster.nufa enable on
>
> Is
>
> gluster volume set gv0 cluster.nufa 1
>
> correct?
>
> Thank you very much!
>
> Ingo
>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
