From: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Monday, October 1, 2018 4:26:03 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 version
-------------------------
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Monday, October 1, 2018 4:26:03 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 version
Dear Ashish,
thank you very much.
I will wait for Nithya and DHT team opinion.
I think I will follow your suggestion changing the names of the new bricks.
Unfortunately, I think that I can’t change the names of the old ones, doesn’t it?!
>>>>. Yeh, I don't think you can change the names.
However, once you are done with this excercise, go through this https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/ and see if it will
work. This is reset-brick command which is mainly used if you want to reset the same brick with same name and path but hostname could be different.
However, i think you should not get into this now.
---
Ashish
Mauro
Il giorno 01 ott 2018, alle ore 12:35, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Hi Mauro,My comments are inline.--AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Monday, October 1, 2018 3:35:26 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionGood morning Ashish,your explanations are always very useful, thank you very much: I will remember these suggestions for any future needs.Anyway, during the week-end, the remove-brick procedures ended successfully and we were able to free up all bricks defined on server s04, s05 and 6 bricks of 12 on server s06.So, we can say that, thanks to your suggestions, we are about to complete this first phase (removing of all bricks defined on s04, s05 and s06 servers).I really appreciated your support.Now I have a last question (I hope): after remove-brick commit I noticed that some data remain on each brick (about 1.2GB of data).Please, take a look to the “df-h_on_s04_s05_s06.txt”.The situation is almost the same on all 3 servers mentioned above: a long list of directories names and some files that are still on the brick, but respective size is 0.Examples:a lot of empty directories on /gluster/mnt*/brick/.glusterfs8 /gluster/mnt2/brick/.glusterfs/b7/1b0 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee94a5-a77c-4c02-85a5-085992840c830 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee85d4-ce48-43a7-a89a-69c728ee8273some empty files in directories in /gluster/mnt*/brick/*[root@s04 ~]# cd /gluster/mnt1/brick/[root@s04 brick]# ls -ltotale 32drwxr-xr-x 7 root root 100 11 set 22.14 archive_calypso[root@s04 brick]# cd archive_calypso/[root@s04 archive_calypso]# lltotale 0drwxr-x--- 3 root 5200 29 11 set 22.13 ans002drwxr-x--- 3 5104 5100 32 11 set 22.14 ans004drwxr-x--- 3 4506 4500 31 11 set 22.14 ans006drwxr-x--- 3 4515 4500 28 11 set 22.14 ans015drwxr-x--- 4 4321 4300 54 11 set 22.14 ans021[root@s04 archive_calypso]# du -a *0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0/echam5/echam_sf006_198110.01.gz0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0/echam50 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.00 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5/echam_sf006_198105.01.gz0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5/echam_sf006_198109.01.gz8 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5What we have to do with this data? Should I backup this “empty” dirs and files on a different storage before deleting them?>>>>> As per my understanding, if remove-bricks was successful then this could be deleted. However, I would like to have thoughts from dht team member on this.@Nithya,Could you please check if this can be deleted?-------------------As soon as all the bricks will be empty, I plan to re-add the new bricks using the following commands:gluster peer detach s04gluster peer detach s05gluster peer detach s06gluster peer probe s04gluster peer probe s05gluster peer probe s06gluster volume add-brick tier2 s04-stg:/gluster/mnt1/brick s05-stg:/gluster/mnt1/brick s06-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s05-stg:/gluster/mnt2/brick s06-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s05-stg:/gluster/mnt3/brick s06-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s05-stg:/gluster/mnt4/brick s06-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s05-stg:/gluster/mnt5/brick s06-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick s05-stg:/gluster/mnt6/brick s06-stg:/gluster/mnt6/brick s04-stg:/gluster/mnt7/brick s05-stg:/gluster/mnt7/brick s06-stg:/gluster/mnt7/brick s04-stg:/gluster/mnt8/brick s05-stg:/gluster/mnt8/brick s06-stg:/gluster/mnt8/brick s04-stg:/gluster/mnt9/brick s05-stg:/gluster/mnt9/brick s06-stg:/gluster/mnt9/brick s04-stg:/gluster/mnt10/brick s05-stg:/gluster/mnt10/brick s06-stg:/gluster/mnt10/brick s04-stg:/gluster/mnt11/brick s05-stg:/gluster/mnt11/brick s06-stg:/gluster/mnt11/brick s04-stg:/gluster/mnt12/brick s05-stg:/gluster/mnt12/brick s06-stg:/gluster/mnt12/brick force>>> This command looks good to me. You have given brick path from s04 to s06 one by one which is right thing to do.However, I would like you to give more meaningful name to bricks associated to one ec subvolume. This will help you in future to find out which brick belongs to which ec sub volume.For example:gluster volume add-brick tier2s04-stg:/gluster/mnt1/brick-13s05-stg:/gluster/mnt1/brick-13s06-stg:/gluster/mnt1/brick-13s04-stg:/gluster/mnt2/brick-13s05-stg:/gluster/mnt2/brick-13s06-stg:/gluster/mnt2/brick-13-13 will indicate that these bricks belongs to 13th ec subvol. You can give any other name also.s04-stg:/gluster/mnt3/brick-14s05-stg:/gluster/mnt3/brick-14s06-stg:/gluster/mnt3/brick-14s04-stg:/gluster/mnt4/brick-14s05-stg:/gluster/mnt4/brick-14s06-stg:/gluster/mnt4/brick-14gluster volume rebalance tier2 fix-layout startgluster volume rebalance tier2 start>>> This looks fine.From your point of view, are they the right commands to close this repairing task?Thank you very much for your help.Regards,MauroIl giorno 01 ott 2018, alle ore 09:17, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Ohh!! It is because brick-multiplexing is "ON" on your setup. Not sure if it is by default ON for 3.12.14 or not.See "cluster.brick-multiplex: on" in gluster v <volname> infoIf brick multiplexing is ON, you will see only one process running for all the bricks on a Node.So we have to do following step to kill any one brick on a node.Steps to kill a brick when multiplex is on -Step - 1Find unix domain_socket of the process on a node.Run "ps -aef | grep glusterfsd" on a node. Example :This is on my machine when I have all the bricks on same machine[root@apandey glusterfs]# ps -aef | grep glusterfsd | grep -v mnt
root 28311 1 0 11:16 ? 00:00:06 /usr/local/sbin/glusterfsd -s apandey --volfile-id vol.apandey.home-apandey-bricks-gluster-vol-1 -p /var/run/gluster/vols/vol/apandey-home-apandey-bricks-gluster-vol-1.pid -S /var/run/gluster/1259033d2ff4f4e5.socket --brick-name /home/apandey/bricks/gluster/vol-1 -l /var/log/glusterfs/bricks/home-apandey-bricks-gluster-vol-1.log --xlator-option *-posix.glusterd-uuid=61b4524c-ccf3-4219-aaff-b3497ac6dd24 --process-name brick --brick-port 49158 --xlator-option vol-server.listen-port=49158Here, /var/run/gluster/1259033d2ff4f4e5.socket is the unix domain socketStep - 2Run following command to kill a brick on the same node -gf_attach -d <unix domain_socket> brick_path_on_that_nodeExample:gf_attach -d /var/run/gluster/1259033d2ff4f4e5.socket /home/apandey/bricks/gluster/vol-6Status of volume: vol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick apandey:/home/apandey/bricks/gluster/
vol-1 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-2 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-3 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-4 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-5 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-6 49158 0 Y 28311
Self-heal Daemon on localhost N/A N/A Y 29787
Task Status of Volume vol
------------------------------------------------------------------------------
There are no active volume tasks
[root@apandey glusterfs]#
[root@apandey glusterfs]#
[root@apandey glusterfs]# gf_attach -d /var/run/gluster/1259033d2ff4f4e5.socket /home/apandey/bricks/gluster/vol-6
OK
[root@apandey glusterfs]# gluster v status
Status of volume: vol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick apandey:/home/apandey/bricks/gluster/
vol-1 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-2 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-3 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-4 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-5 49158 0 Y 28311
Brick apandey:/home/apandey/bricks/gluster/
vol-6 N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y 29787
Task Status of Volume vol
------------------------------------------------------------------------------
There are no active volume tasksTo start a brick we just need to start volume using "force"gluster v start <volname> force----AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, September 28, 2018 9:25:53 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionI asked you how to detect the PID of a specific brick because I see that more than one brick has the same PID (also on my virtual env).If I kill one of them I risk to kill some other brick. Is it normal?[root@s01 ~]# gluster vol statusStatus of volume: tier2Gluster process TCP Port RDMA Port Online Pid------------------------------------------------------------------------------Brick s01-stg:/gluster/mnt1/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt1/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt1/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt2/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt2/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt2/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt3/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt3/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt3/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt4/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt4/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt4/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt5/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt5/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt5/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt6/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt6/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt6/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt7/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt7/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt7/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt8/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt8/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt8/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt9/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt9/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt9/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt10/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt10/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt10/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt11/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt11/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt11/brick 49153 0 Y 3953Brick s01-stg:/gluster/mnt12/brick 49153 0 Y 3956Brick s02-stg:/gluster/mnt12/brick 49153 0 Y 3956Brick s03-stg:/gluster/mnt12/brick 49153 0 Y 3953Brick s04-stg:/gluster/mnt1/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt2/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt3/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt4/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt5/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt6/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt7/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt8/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt9/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt10/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt11/brick 49153 0 Y 3433Brick s04-stg:/gluster/mnt12/brick 49153 0 Y 3433Brick s05-stg:/gluster/mnt1/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt2/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt3/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt4/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt5/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt6/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt7/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt8/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt9/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt10/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt11/brick 49153 0 Y 3709Brick s05-stg:/gluster/mnt12/brick 49153 0 Y 3709Brick s06-stg:/gluster/mnt1/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt2/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt3/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt4/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt5/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt6/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt7/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt8/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt9/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt10/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt11/brick 49153 0 Y 3644Brick s06-stg:/gluster/mnt12/brick 49153 0 Y 3644Self-heal Daemon on localhost N/A N/A Y 79376Quota Daemon on localhost N/A N/A Y 79472Bitrot Daemon on localhost N/A N/A Y 79485Scrubber Daemon on localhost N/A N/A Y 79505Self-heal Daemon on s03-stg N/A N/A Y 77073Quota Daemon on s03-stg N/A N/A Y 77148Bitrot Daemon on s03-stg N/A N/A Y 77160Scrubber Daemon on s03-stg N/A N/A Y 77191Self-heal Daemon on s02-stg N/A N/A Y 80150Quota Daemon on s02-stg N/A N/A Y 80226Bitrot Daemon on s02-stg N/A N/A Y 80238Scrubber Daemon on s02-stg N/A N/A Y 80269Self-heal Daemon on s04-stg N/A N/A Y 106815Quota Daemon on s04-stg N/A N/A Y 106866Bitrot Daemon on s04-stg N/A N/A Y 106878Scrubber Daemon on s04-stg N/A N/A Y 106897Self-heal Daemon on s05-stg N/A N/A Y 130807Quota Daemon on s05-stg N/A N/A Y 130884Bitrot Daemon on s05-stg N/A N/A Y 130896Scrubber Daemon on s05-stg N/A N/A Y 130927Self-heal Daemon on s06-stg N/A N/A Y 157146Quota Daemon on s06-stg N/A N/A Y 157239Bitrot Daemon on s06-stg N/A N/A Y 157252Scrubber Daemon on s06-stg N/A N/A Y 157288Task Status of Volume tier2------------------------------------------------------------------------------Task : Remove brickID : 06ec63bb-a441-4b85-b3cf-ac8e9df4830fRemoved bricks:s04-stg:/gluster/mnt1/bricks04-stg:/gluster/mnt2/bricks04-stg:/gluster/mnt3/bricks04-stg:/gluster/mnt4/bricks04-stg:/gluster/mnt5/bricks04-stg:/gluster/mnt6/brickStatus : in progress[root@s01 ~]# ps -ef|grep glusterfsroot 3956 1 79 set25 ? 2-14:33:57 /usr/sbin/glusterfsd -s s01-stg --volfile-id tier2.s01-stg.gluster-mnt1-brick -p /var/run/gluster/vols/tier2/s01-stg-gluster-mnt1-brick.pid -S /var/run/gluster/a889b8a21ac2afcbfa0563b9dd4db265.socket --brick-name /gluster/mnt1/brick -l /var/log/glusterfs/bricks/gluster-mnt1-brick.log --xlator-option *-posix.glusterd-uuid=b734b083-4630-4523-9402-05d03565efee --brick-port 49153 --xlator-option tier2-server.listen-port=49153root 79376 1 0 09:16 ? 00:04:16 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/4fab1a27e6ee700b3b9a3b3393ab7445.socket --xlator-option *replicate*.node-uuid=b734b083-4630-4523-9402-05d03565efeeroot 79472 1 0 09:16 ? 00:00:42 /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/run/gluster/quotad/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/gluster/958ab34799fc58f4dfe20e5732eea70b.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=offroot 79485 1 7 09:16 ? 00:40:43 /usr/sbin/glusterfs -s localhost --volfile-id gluster/bitd -p /var/run/gluster/bitd/bitd.pid -l /var/log/glusterfs/bitd.log -S /var/run/gluster/b2ea9da593fae1bc4d94e65aefdbdda9.socket --global-timer-wheelroot 79505 1 0 09:16 ? 00:00:01 /usr/sbin/glusterfs -s localhost --volfile-id gluster/scrub -p /var/run/gluster/scrub/scrub.pid -l /var/logglusterfs/scrub.log -S /var/run/gluster/ee7886cbcf8d2adf261084b608c905d5.socket --global-timer-wheelroot 137362 137225 0 17:53 pts/0 00:00:00 grep --color=auto glusterfsIl giorno 28 set 2018, alle ore 17:47, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:From: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "Gluster Users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, September 28, 2018 9:08:52 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionThank you, Ashish.I will study and try your solution on my virtual env.How I can detect the process of a brick on gluster server?Many Thanks,Maurogluster v status <volname> will give you the list of bricks and the respective process id.Also, you can use "ps aux | grep glusterfs" to see all the processes on a node but I think the above step also do the same.---AshishIl ven 28 set 2018 16:39 Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:From: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, September 28, 2018 7:08:41 PM
Subject: Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 versionDear Ashish,please excuse me, I'm very sorry for misunderstanding.Before contacting you during last days, we checked all network devices (switch 10GbE, cables, NICs, servers ports, and so on), operating systems version and settings, network bonding configuration, gluster packages versions, tuning profiles, etc. but everything seems to be ok. The first 3 servers (and volume) operated without problem for one year. After we added the new 3 servers we noticed something wrong.Fortunately, yesterday you gave me an hand to understand where is (or could be) the problem.At this moment, after we re-launched the remove-brick command, it seems that the rebalance is going ahead without errors, but it is only scanning the files.May be that during the future data movement some errors could appear.For this reason, it could be useful to know how to proceed in case of a new failure: insist with approach n.1 or change the strategy?We are thinking to try to complete the running remove-brick procedure and make a decision based on the outcome.Question: could we start approach n.2 also after having successfully removed the V1 subvolume?!>>> Yes, we can do that. My idea is to use replace-brick command.We will kill "ONLY" one brick process on s06. We will format this brick. Then use replace-brick command to replace brick of a volume on s05 with this formatted brick.heal will be triggered and data of the respective volume will be placed on this brick.Now, we can format the brick which got freed up on s05 and replace the brick which we killed on s06 to s05.During this process, we have to make sure heal completed before trying any other replace/kill brick.It is tricky but looks doable. Think about it and try to perform it on your virtual environment first before trying on production.-------If it is still possible, could you please illustrate the approach n.2 even if I dont have free disks?I would like to start thinking about it and test it on a virtual environment.Thank you in advance for your help and patience.Regards,MauroIl giorno 28 set 2018, alle ore 14:36, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:
We could have taken approach -2 even if you did not have free disks. You should have told me why are you
opting Approach-1 or perhaps I should have asked.
I was wondering for approach 1 because sometimes re-balance takes time depending upon the data size.Anyway, I hope whole setup is stable, I mean it is not in the middle of something which we can not stop.
If free disks are the only concern I will give you some more steps to deal with it and follow the approach 2.Let me know once you think everything is fine with the system and there is nothing to heal.---
AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Friday, September 28, 2018 4:21:03 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionHi Ashish,as I said in my previous message, we adopted the first approach you suggested (setting network.ping-timeout option to 0).This choice was due to the absence of empty brick to be used as indicated in the second approach.So, we launched remove-brick command on the first subvolume (V1, bricks 1,2,3,4,5,6 on server s04).Rebalance started moving the data across the other bricks, but, after about 3TB of moved data, rebalance speed slowed down and some transfer errors appeared in the rebalance.log of server s04.At this point, since remaining 1,8TB need to be moved in order to complete the step, we decided to stop the remove-brick execution and start it again (I hope it doesn’t stop again before complete the rebalance)Now rebalance is not moving data, it’s only scanning files (please, take a look to the following output)[root@s01 ~]# gluster volume remove-brick tier2 s04-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick statusNode Rebalanced-files size scanned failures skipped status run time in h:m:s--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------s04-stg 0 0Bytes 182008 0 0 in progress 3:08:09Estimated time left for rebalance to complete : 442:45:06If I’m not wrong, remove-brick rebalances entire cluster each time it start.Is there a way to speed up this procedure? Do you have some other suggestion that, in this particular case, could be useful to reduce errors (I know that they are related to the current volume configuration) and improve rebalance performance avoiding to rebalance the entire cluster?Thank you in advance,MauroIl giorno 27 set 2018, alle ore 13:14, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Yes, you can.If not me others may also reply.---AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Thursday, September 27, 2018 4:24:12 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionDear Ashish,I can not thank you enough!Your procedure and description is very detailed.I think to follow the first approach after setting network.ping-timeout option to 0 (If I’m not wrong “0" means “infinite”...I noticed that this value reduced rebalance errors).After the fix I will set network.ping-timeout option to default value.Could I contact you again if I need some kind of suggestion?Thank you very much again.Have a good day,MauroIl giorno 27 set 2018, alle ore 12:38, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Hi Mauro,We can divide the 36 newly added bricks into 6 set of 6 bricks each starting from brick37.That means, there are 6 ec subvolumes and we have to deal with one sub volume at a time.I have named it V1 to V6.Problem:Take the case of V1.The best configuration/setup would be to have all the 6 bricks of V1 on 6 different nodes.However, in your case you have added 3 new nodes. So, at least we should have 2 bricks on 3 different newly added nodes.This way, in 4+2 EC configuration, even if one node goes down you will have 4 other bricks of that volume and the data on that volume would be accessible.In current setup if s04-stg goes down, you will loose all the data on V1 and V2 as all the bricks will be down. We want to avoid and correct it.Now, we can have two approach to correct/modify this setup.Approach 1We have to remove all the newly added bricks in a set of 6 bricks. This will trigger re- balance and move whole data to other sub volumes.Repeat the above step and then once all the bricks are removed, add those bricks again in a set of 6 bricks, this time have 2 bricks from each of the 3 newly added Nodes.While this is a valid and working approach, I personally think that this will take long time and also require lot of movement of data.Approach 2In this approach we can use the heal process. We have to deal with all the volumes (V1 to V6) one by one. Following are the steps for V1-Step 1 -Use replace-brick command to move following bricks on s05-stg node one by one (heal should be completed after every replace brick command)Brick39: s04-stg:/gluster/mnt3/brick to s05-stg/<brick which is free>Brick40: s04-stg:/gluster/mnt4/brick to s05-stg/<other brick which is free>Command :gluster v replace-brick <volname> s04-stg:/gluster/mnt3/brick s05-stg:/<brick which is free> commit forceTry to give names to the bricks so that you can identify which 6 bricks belongs to same ec subvolumeUse replace-brick command to move following bricks on s06-stg node one by oneBrick41: s04-stg:/gluster/mnt5/brick to s06-stg/<brick which is free>Brick42: s04-stg:/gluster/mnt6/brick to s06-stg/<other brick which is free>Step 2 - After, every replace-brick command, you have to wait for heal to be completed.check "gluster v heal <volname> info " if it shows any entry you have to wait for it to be completed.After successful step 1 and step 2, setup for sub volume V1 will be fixed. The same steps you have to perform for other volumes. Only thing is thatthe nodes would be different on which you have to move the bricks.V1Brick37: s04-stg:/gluster/mnt1/brickBrick38: s04-stg:/gluster/mnt2/brickBrick39: s04-stg:/gluster/mnt3/brickBrick40: s04-stg:/gluster/mnt4/brickBrick41: s04-stg:/gluster/mnt5/brickBrick42: s04-stg:/gluster/mnt6/brickV2Brick43: s04-stg:/gluster/mnt7/brickBrick44: s04-stg:/gluster/mnt8/brickBrick45: s04-stg:/gluster/mnt9/brickBrick46: s04-stg:/gluster/mnt10/brickBrick47: s04-stg:/gluster/mnt11/brickBrick48: s04-stg:/gluster/mnt12/brickV3Brick49: s05-stg:/gluster/mnt1/brickBrick50: s05-stg:/gluster/mnt2/brickBrick51: s05-stg:/gluster/mnt3/brickBrick52: s05-stg:/gluster/mnt4/brickBrick53: s05-stg:/gluster/mnt5/brickBrick54: s05-stg:/gluster/mnt6/brickV4Brick55: s05-stg:/gluster/mnt7/brickBrick56: s05-stg:/gluster/mnt8/brickBrick57: s05-stg:/gluster/mnt9/brickBrick58: s05-stg:/gluster/mnt10/brickBrick59: s05-stg:/gluster/mnt11/brickBrick60: s05-stg:/gluster/mnt12/brickV5Brick61: s06-stg:/gluster/mnt1/brickBrick62: s06-stg:/gluster/mnt2/brickBrick63: s06-stg:/gluster/mnt3/brickBrick64: s06-stg:/gluster/mnt4/brickBrick65: s06-stg:/gluster/mnt5/brickBrick66: s06-stg:/gluster/mnt6/brickV6Brick67: s06-stg:/gluster/mnt7/brickBrick68: s06-stg:/gluster/mnt8/brickBrick69: s06-stg:/gluster/mnt9/brickBrick70: s06-stg:/gluster/mnt10/brickBrick71: s06-stg:/gluster/mnt11/brickBrick72: s06-stg:/gluster/mnt12/brickJust a note that these steps need movement of data.Be careful while performing these steps and do one replace brick at a time and only after heal completion go to next.Let me know if you have any issues.---AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Thursday, September 27, 2018 4:03:04 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionDear Ashish,I hope I don’t disturb you so much, but I would like to ask you if you had some time to dedicate to our problem.Please, forgive my insistence.Thank you in advance,MauroIl giorno 26 set 2018, alle ore 19:56, Mauro Tridici <mauro.tridici@xxxxxxx> ha scritto:Hi Ashish,sure, no problem! We are a little bit worried, but we can wait :-)Thank you very much for your support and your availability.Regards,MauroIl giorno 26 set 2018, alle ore 19:33, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Hi Mauro,Yes, I can provide you step by step procedure to correct it.Is it fine If i provide you the steps tomorrow as it is quite late over here and I don't want to miss anything in hurry?---AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Wednesday, September 26, 2018 6:54:19 PM
Subject: Re: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionHi Ashish,in attachment you can find the rebalance log file and the last updated brick log file (the other files in /var/log/glusterfs/bricks directory seem to be too old).I just stopped the running rebalance (as you can see at the bottom of the rebalance log file).So, if exists a safe procedure to correct the problem I would like execute it.I don’t know if I can ask you it, but, if it is possible, could you please describe me step by step the right procedure to remove the newly added bricks without losing the data that have been already rebalanced?The following outputs show the result of “df -h” command executed on one of the first 3 nodes (s01, s02, s03) already existing and on one of the last 3 nodes (s04, s05, s06) added recently.[root@s06 bricks]# df -hFile system Dim. Usati Dispon. Uso% Montato su/dev/mapper/cl_s06-root 100G 2,1G 98G 3% /devtmpfs 32G 0 32G 0% /devtmpfs 32G 4,0K 32G 1% /dev/shmtmpfs 32G 26M 32G 1% /runtmpfs 32G 0 32G 0% /sys/fs/cgroup/dev/mapper/cl_s06-var 100G 2,0G 99G 2% /var/dev/mapper/cl_s06-gluster 100G 33M 100G 1% /gluster/dev/sda1 1014M 152M 863M 15% /boot/dev/mapper/gluster_vgd-gluster_lvd 9,0T 807G 8,3T 9% /gluster/mnt3/dev/mapper/gluster_vgg-gluster_lvg 9,0T 807G 8,3T 9% /gluster/mnt6/dev/mapper/gluster_vgc-gluster_lvc 9,0T 807G 8,3T 9% /gluster/mnt2/dev/mapper/gluster_vge-gluster_lve 9,0T 807G 8,3T 9% /gluster/mnt4/dev/mapper/gluster_vgj-gluster_lvj 9,0T 887G 8,2T 10% /gluster/mnt9/dev/mapper/gluster_vgb-gluster_lvb 9,0T 807G 8,3T 9% /gluster/mnt1/dev/mapper/gluster_vgh-gluster_lvh 9,0T 887G 8,2T 10% /gluster/mnt7/dev/mapper/gluster_vgf-gluster_lvf 9,0T 807G 8,3T 9% /gluster/mnt5/dev/mapper/gluster_vgi-gluster_lvi 9,0T 887G 8,2T 10% /gluster/mnt8/dev/mapper/gluster_vgl-gluster_lvl 9,0T 887G 8,2T 10% /gluster/mnt11/dev/mapper/gluster_vgk-gluster_lvk 9,0T 887G 8,2T 10% /gluster/mnt10/dev/mapper/gluster_vgm-gluster_lvm 9,0T 887G 8,2T 10% /gluster/mnt12tmpfs 6,3G 0 6,3G 0% /run/user/0[root@s01 ~]# df -hFile system Dim. Usati Dispon. Uso% Montato su/dev/mapper/cl_s01-root 100G 5,3G 95G 6% /devtmpfs 32G 0 32G 0% /devtmpfs 32G 39M 32G 1% /dev/shmtmpfs 32G 26M 32G 1% /runtmpfs 32G 0 32G 0% /sys/fs/cgroup/dev/mapper/cl_s01-var 100G 11G 90G 11% /var/dev/md127 1015M 151M 865M 15% /boot/dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster/dev/mapper/gluster_vgi-gluster_lvi 9,0T 5,5T 3,6T 61% /gluster/mnt7/dev/mapper/gluster_vgm-gluster_lvm 9,0T 5,4T 3,6T 61% /gluster/mnt11/dev/mapper/gluster_vgf-gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4/dev/mapper/gluster_vgl-gluster_lvl 9,0T 5,8T 3,3T 64% /gluster/mnt10/dev/mapper/gluster_vgj-gluster_lvj 9,0T 5,5T 3,6T 61% /gluster/mnt8/dev/mapper/gluster_vgn-gluster_lvn 9,0T 5,4T 3,6T 61% /gluster/mnt12/dev/mapper/gluster_vgk-gluster_lvk 9,0T 5,8T 3,3T 64% /gluster/mnt9/dev/mapper/gluster_vgh-gluster_lvh 9,0T 5,6T 3,5T 63% /gluster/mnt6/dev/mapper/gluster_vgg-gluster_lvg 9,0T 5,6T 3,5T 63% /gluster/mnt5/dev/mapper/gluster_vge-gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3/dev/mapper/gluster_vgc-gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1/dev/mapper/gluster_vgd-gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2tmpfs 6,3G 0 6,3G 0% /run/user/0s01-stg:tier2 420T 159T 262T 38% /tier2As you can see, used space value of each brick of the last servers is about 800GB.Thank you,MauroIl giorno 26 set 2018, alle ore 14:51, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:Hi Mauro,rebalance and brick logs should be the first thing we should go through.There is a procedure to correct the configuration/setup but the situation you are in is difficult to follow that procedure.You should have added the bricks hosted on s04-stg, s05-stg and s06-stg the same way you had the previous configuration.That means 2 bricks on each node for one subvolume.The procedure will require a lot of replace bricks which will again need healing and all. In addition to that we have to wait for re-balance to complete.I would suggest that if whole data has not been rebalanced and if you can stop the rebalance and remove these newly added bricks properly then you should remove these newly added bricks.After that, add these bricks so that you have 2 bricks of each volume on 3 newly added nodes.Yes, it is like undoing whole effort but it is better to do it now then facing issues in future when it will be almost impossible to correct these things if you have lots of data.---AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "Ashish Pandey" <aspandey@xxxxxxxxxx>
Cc: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Wednesday, September 26, 2018 5:55:02 PM
Subject: Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 versionDear Ashish,thank you for you answer.I could provide you the entire log file related to glusterd, glusterfsd and rebalance.Please, could you indicate which one you need first?Yes, we added the last 36 bricks after creating vol. Is there a procedure to correct this error? Is it still possible to do it?Many thanks,MauroIl giorno 26 set 2018, alle ore 14:13, Ashish Pandey <aspandey@xxxxxxxxxx> ha scritto:I think we don't have enough logs to debug this so I would suggest you to provide more logs/info.I have also observed that the configuration and setup of your volume is not very efficient.For example:Brick37: s04-stg:/gluster/mnt1/brickBrick38: s04-stg:/gluster/mnt2/brickBrick39: s04-stg:/gluster/mnt3/brickBrick40: s04-stg:/gluster/mnt4/brickBrick41: s04-stg:/gluster/mnt5/brickBrick42: s04-stg:/gluster/mnt6/brickBrick43: s04-stg:/gluster/mnt7/brickBrick44: s04-stg:/gluster/mnt8/brickBrick45: s04-stg:/gluster/mnt9/brickBrick46: s04-stg:/gluster/mnt10/brickBrick47: s04-stg:/gluster/mnt11/brickBrick48: s04-stg:/gluster/mnt12/brickThese 12 bricks are on same node and the sub volume made up of these bricks will be of same subvolume, which is not good. Same is true for the bricks hosted on s05-stg and s06-stgI think you have added these bricks after creating vol. The probability of disruption in connection of these bricks will be higher in this case.---AshishFrom: "Mauro Tridici" <mauro.tridici@xxxxxxx>
To: "gluster-users" <gluster-users@xxxxxxxxxxx>
Sent: Wednesday, September 26, 2018 3:38:35 PM
Subject: Rebalance failed on Distributed Disperse volume based on 3.12.14 versionDear All, Dear Nithya,after upgrading from 3.10.5 version to 3.12.14, I tried to start a rebalance process to distribute data across the bricks, but something goes wrong.Rebalance failed on different nodes and the time value needed to complete the procedure seems to be very high.[root@s01 ~]# gluster volume rebalance tier2 statusNode Rebalanced-files size scanned failures skipped status run time in h:m:s--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------localhost 19 161.6GB 537 2 2 in progress 0:32:23s02-stg 25 212.7GB 526 5 2 in progress 0:32:25s03-stg 4 69.1GB 511 0 0 in progress 0:32:25s04-stg 4 484Bytes 12283 0 3 in progress 0:32:25s05-stg 23 484Bytes 11049 0 10 in progress 0:32:25s06-stg 3 1.2GB 8032 11 3 failed 0:17:57Estimated time left for rebalance to complete : 3601:05:41volume rebalance: tier2: successWhen rebalance processes fail, I can see the following kind of errors in /var/log/glusterfs/tier2-rebalance.logError type 1)[2018-09-26 08:50:19.872575] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-10: Operation failed on 2 of 6 subvolumes.(up=111111, mask=100111, remaining=000000, good=100111, bad=011000)[2018-09-26 08:50:19.901792] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-11: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111101, remaining=000000, good=111101, bad=000010)Error type 2)[2018-09-26 08:53:31.566836] W [socket.c:600:__socket_rwv] 0-tier2-client-53: readv on 192.168.0.55:49153 failed (Connection reset by peer)Error type 3)[2018-09-26 08:57:37.852590] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)[2018-09-26 08:57:39.282306] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)[2018-09-26 09:02:04.928408] W [MSGID: 109023] [dht-rebalance.c:1013:__dht_check_free_space] 0-tier2-dht: data movement of file {blocks:0 name:(/OPA/archive/historical/dts/MREA/Observations/Observations/MREA14/Cs-1/CMCC/raw/CS013.ext)} would result in dst node (tier2-disperse-5:2440190848) having lower disk space than the source node (tier2-disperse-11:71373083776).Skipping file.Error type 4)W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-tier2-client-7: socket disconnectedError type 5)[2018-09-26 09:07:42.333720] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f0417e0ee25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5590086004b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55900860032b] ) 0-: received signum (15), shutting downError type 6)[2018-09-25 08:09:18.340658] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-4: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting.It seems that there are some network or timeout problems, but the network usage/traffic values are not so high.Do you think that, in my volume configuration, I have to modify some volume options related to thread and/or network parameters?Could you, please, help me to understand the cause of the problems above?You can find below our volume info:(volume is implemented on 6 servers; each server configuration: 2 cpu 10-cores, 64GB RAM, 1 SSD dedicated to the OS, 12 x 10TB HD)[root@s04 ~]# gluster vol info<div style="margin: 0px; line-height: nor
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------------------
Mauro Tridici
Fondazione CMCC
CMCC Supercomputing Center
presso Complesso Ecotekne - Università del Salento -
Strada Prov.le Lecce - Monteroni sn
73100 Lecce IT
mobile: (+39) 327 5630841
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
