Hi Vlad, because I never got affinity and distributed.migrate-data using attrs working, I finally invested time in nufa and found out that (against docs) that "gluster volume set gv0 cluster.nufa 1" works to enable nufa, so I did also with the data I had. Then I started to copy files off the storage and then copied them back on the correct host and so re-distributed them as I liked it to be. It would have been easier if setting a "custom" affinity and such would have worked but now I reached my goal. Ingo Am 31.10.18 um 04:57 schrieb Vlad Kopylov: > nufa helps you write to local brick, if replication is involved it will > still copy it to other bricks (or suppose to do so) > what might be happening is that when initial file was created other > nodes were down and it didn't replicate properly and now heal is failing > check your > gluster vol heal Volname info > > maybe you will find out where second copy of the file suppose to be - > and just copy it to that brick > > On Sun, Oct 28, 2018 at 6:07 PM Ingo Fischer <nl@xxxxxxxxxxxxx > <mailto:nl@xxxxxxxxxxxxx>> wrote: > > Hi All, > > has noone an idea on system.affinity/distributed.migrate-data ? > Or how to correctly enable nufa? > > BTW: the used gluster version is 4.1.5 > > Thank you for your help on this! > > Ingo > > Am 24.10.18 um 12:54 schrieb Ingo Fischer: > > Hi, > > > > I have setup a glusterfs volume gv0 as distributed/replicated: > > > > root@pm1:~# gluster volume info gv0 > > > > Volume Name: gv0 > > Type: Distributed-Replicate > > Volume ID: 64651501-6df2-4106-b330-fdb3e1fbcdf4 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 3 x 2 = 6 > > Transport-type: tcp > > Bricks: > > Brick1: 192.168.178.50:/gluster/brick1/gv0 > > Brick2: 192.168.178.76:/gluster/brick1/gv0 > > Brick3: 192.168.178.50:/gluster/brick2/gv0 > > Brick4: 192.168.178.81:/gluster/brick1/gv0 > > Brick5: 192.168.178.50:/gluster/brick3/gv0 > > Brick6: 192.168.178.82:/gluster/brick1/gv0 > > Options Reconfigured: > > performance.client-io-threads: off > > nfs.disable: on > > transport.address-family: inet > > > > > > root@pm1:~# gluster volume status > > Status of volume: gv0 > > Gluster process TCP Port RDMA Port > Online Pid > > > ------------------------------------------------------------------------------ > > Brick 192.168.178.50:/gluster/brick1/gv0 49152 0 Y > > 1665 > > Brick 192.168.178.76:/gluster/brick1/gv0 49152 0 Y > > 26343 > > Brick 192.168.178.50:/gluster/brick2/gv0 49153 0 Y > > 1666 > > Brick 192.168.178.81:/gluster/brick1/gv0 49152 0 Y > > 1161 > > Brick 192.168.178.50:/gluster/brick3/gv0 49154 0 Y > > 1679 > > Brick 192.168.178.82:/gluster/brick1/gv0 49152 0 Y > > 1334 > > Self-heal Daemon on localhost N/A N/A Y > > 5022 > > Self-heal Daemon on 192.168.178.81 N/A N/A Y > > 935 > > Self-heal Daemon on 192.168.178.82 N/A N/A Y > > 1057 > > Self-heal Daemon on pm2.fritz.box N/A N/A Y > > 1651 > > > > > > I use the fs to store VM files, so not many, but big files. > > > > The distribution now put 4 big files on one brick set and only one > file > > on an other. This means that the one brick set it "overcommited" > now as > > soon as all VMs using max space. SO I would like to manually > > redistribute the files a bit better. > > > > After log googling I found that the following should work: > > setfattr -n 'system.affinity' -v $location $filepath > > setfattr -n 'distribute.migrate-data' -v 'force' $filepath > > > > But I have problems with it because it gives errors or doing > nothing at all. > > > > The mounting looks like: > > 192.168.178.50:gv0 on /mnt/pve/glusterfs type fuse.glusterfs > > > (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > > > Here is what I tried for the first xattr: > > > > root@pm1:~# setfattr -n 'system.affinity' -v 'gv0-client-5' > > /mnt/pve/glusterfs/201/imagesvm201.qcow2 > > setfattr: /mnt/pve/glusterfs/201/imagesvm201.qcow2: Operation not > supported > > > > So I found on google to use trusted.affinity instead and yes this > works. > > I'm only not sure if the location "gv0-client-5" is correct to > move the > > file to "Brick 5" from "gluster volume info gv0" ... or how this > > location is build? > > Commit Message from > http://review.gluster.org/#/c/glusterfs/+/5233/ says > >> The value is the internal client or AFR brick name where you want the > > file to be. > > > > So what do I need to set there? maybe I do need the "afr" because > > replicated? But where to get that name from? > > I also tried to enter other client or replicate names like > > "gv0-replicate-0" or such which seems to be more fitting for a > > replicated volume, but result the same. > > > > > > For the second command I get: > > root@pm1:~# setfattr -n 'distribute.migrate-data' -v 'force' > > /mnt/pve/glusterfs/201/imagesvm201.qcow2 > > setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: Operation > > not supported > > root@pm1:~# setfattr -n 'trusted.distribute.migrate-data' -v 'force' > > /mnt/pve/glusterfs/201/imagesvm201.qcow2 > > setfattr: /mnt/pve/glusterfs/images/201/vm-201-disk-0.qcow2: File > exists > > > > I also experimented with other "names" then "gv0-client-5" above but > > always the same. > > > > I saw that instead of the second command I could start a rebalance > with > > force, but this also did nothing. Ended after max1 second and > moved nothing. > > > > Can someone please advice how to do it right? > > > > > > An other idea was to enable nufa and kind of "re-copy" the files > on the > > glusterfs, but here it seems that the documentation is wrong. > > gluster volume set gv0 cluster.nufa enable on > > > > Is > > > > gluster volume set gv0 cluster.nufa 1 > > > > correct? > > > > Thank you very much! > > > > Ingo > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx <mailto:Gluster-users@xxxxxxxxxxx> > https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
