Long post about gluster issue after 6.5 to 6.6 upgrade and recovery steps

[Strahil Nikolov <hunter86_bg@xxxxxxxxx>]

Hello Colleagues,

I want to share my experience and especially how I have recovered after a situation where gluster refuses to heal several files of my oVirt Lab.

Most probably the situation was caused by the fact that I didn't check if all files were healed before I started the upgrade on the next node , which in a 'replica 2 arbiter1' setup has caused multiple files missing/conflicting and heal fails to happen.

Background:

1. I have powered off my HostedEngine VM and made a snapshot of the gluster volume

2. Started the update , but without screen - I messed it up and decided to revert from the snapshot

3. Powered off the volume , restored from snapshot and then started (and again snapshoted the volume)

4. Upgrade of the HostedEngine was successfull

5. Upgraded the arbiter (ovirt3)

6. I forgot to check the heal status on the arbiter and upgraded ovirt1/gluster1 (which maybe was the reason for the issue)

7. After gluster1 was healed I saw that some files are left for healing , but expected it will finish till ovirt2/gluster2 is patched

8. Sadly my assumption was not right and after the reboot of ovirt2/gluster2 I noticed that some files never heal.

Symptoms:

2 files per volume never heal (used 'full' mode) even after I 'stat'-ed every file/dir in the volume.

Ovirt Dashboard reported multiple errors (50+) that it cannot update the OVF metadata for volume/VM.

Here are my notes while I was recovering from the situation. As this is my LAB, I shutdown all VMs (including the HostedEngine) as downtime was not an issue:

Heals never complete:

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
<gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8>  
<gfid:95bc2cd2-8a1e-464e-a384-6b128780d370>  
Status: Connected
Number of entries: 2

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
<gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8>  
<gfid:95bc2cd2-8a1e-464e-a384-6b128780d370>  
Status: Connected
Number of entries: 2


Mount to get the gfid to path relationship:
mount -t glusterfs -o aux-gfid-mount gluster1:/data_fast4 /mnt

# getfattr -n trusted.glusterfs.pathinfo -e text /mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
getfattr: Removing leading '/' from absolute path names
# file: mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
trusted.glusterfs.pathinfo="(<REPLICATE:data_fast4-replicate-0> <POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt2.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a651
2-eaf6-4859-90cf-eeef2cc0cab8> <POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt3.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8>)"

The local brick (that is supposed to be healed) is missing some data:

# ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
ls: няма достъп до /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8: Няма такъв файл или директория

Remote is OK:
# ssh gluster2 'ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
-rw-r--r--. 2 vdsm kvm 436  9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8

Arbiter is also OK:
# ssh ovirt3 'ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
-rw-r--r--. 2 vdsm kvm 0  9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8


Rsync the file/directory from a good brick to broken one:

# rsync -avP gluster2:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/ /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/
# ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
-rw-r--r--. 1 vdsm kvm 436  9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8

After a full heal we see our problematic file:

# gluster volume heal data_fast4 full
Launching heal operation to perform full self heal on volume data_fast4 has been successful  
Use heal info commands to check status.

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta  
Status: Connected
Number of entries: 1

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta  
Status: Connected
Number of entries: 1


This time the file is missing on both gluster1 (older version) and arbiter:

# cat /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
CTIME=1558265783
DESCRIPTION={"Updated":true,"Size":256000,"Last Updated":"Sat Nov 09 19:24:08 EET 2019","Storage Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk Description":"OVF_STORE"}
DISKTYPE=OVFS
DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
FORMAT=RAW
GEN=0
IMAGE=58e197a6-12df-4432-a643-298d40e44130
LEGALITY=LEGAL
MTIME=0
PUUID=00000000-0000-0000-0000-000000000000
SIZE=262144
TYPE=PREALLOCATED
VOLTYPE=LEAF
EOF

# ssh gluster2 cat /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
CTIME=1558265783
DESCRIPTION={"Updated":true,"Size":256000,"Last Updated":"Sat Nov 09 19:34:33 EET 2019","Storage Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk Description":"OVF_STORE"}
DISKTYPE=OVFS
DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
FORMAT=RAW
GEN=0
IMAGE=58e197a6-12df-4432-a643-298d40e44130
LEGALITY=LEGAL
MTIME=0
PUUID=00000000-0000-0000-0000-000000000000
SIZE=262144
TYPE=PREALLOCATED
VOLTYPE=LEAF
EOF

# ssh ovirt3 cat /578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
cat: /578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta: Няма такъв файл или директория

As we miss the same file on both a brick and arbiter we take another approach:
First remove on gluster1 the file to another name:

# mv /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta /gluster_bricks/data_fast4
/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta_old

Then we copy the file from gluster2 (good brick)
# rsync -avP gluster2:/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta /gluster_
bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
receiving incremental file list
535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
           436 100%  425.78kB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 43 bytes  received 571 bytes  1,228.00 bytes/sec
total size is 436  speedup is 0.71

Run full heal:
# gluster volume heal data_fast4 full
Launching heal operation to perform full self heal on volume data_fast4 has been successful  
Use heal info commands to check status

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0


And ofcourse umount /mnt

I did the above for all oVirt Storage domains and rebooted all nodes (simultaneously) after stopping the whole stack. This one should not be neccessary , but I wanted to be sure that after power outage the cluster will be operational again:

systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd supervdsmd sanlock glusterd

/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh

Verification:

After the reboot , I have tried to set each oVirt storage domain in 'Maintenance' which confirms that engine can update the OVMF meta and then set it back to Active. Without downtime , this will not be possible.

I hope this long post will help anyone .

PS: I have collected some data for some of the files, that I have ommited as this e-mail is very long.

Best Regards,

Strahil Nikolov

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users