This morning I found my gluster volume broken. Whatever 'gluster volume xxxxx gv0' commands I tried, they timed out. The logs were not very helpful. Restarting gluster on all nodes did not help. Actually nothing helped, and I didn't even know what to look for where. The volume spans over three nodes of one brick each, running Centos 7 and 8. Eventually I realised that the 9.2 rpms were released last night, and yum cron had upgraded my 9.1 to 9.2 on two of the nodes. The third one was still running 9.1. I stopped gluster on all of them, downgraded the two nodes back to 9.1, and the problem was solved; the volume came back up just fine. Subsequently I stopped gluster on all nodes, upgraded all of them to 9.2, and restarted; the volume came back up just fine again. Conclusion: something in gluster doesn't like version mismatches. On systems that run automatic updates, gluster should be excluded and only be upgraded manually at the same time across the entire cluster. One question remains: how can a cluster be upgraded without taking it down? Stopping, upgrading, and restarting one node at a time doesn't seem to work. ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
