Hi Atul, In Short: it is due to client side quorum behavior Detailed info: I see that there are 3 nodes in the cluster ie master1, master2, compute01 However the volume is being hosted only on master1 and master2. Also, see that you have enabled server side quorum, and client side quorum from vol info cluster.quorum-type: auto =================>client side quorum options cluster.server-quorum-type: server =================>server side quorum options cluster.server-quorum-ratio: 51% =================>server side quorum options Given that you are using compute01 (which is more of a dummy node in this case), hence even though one node is down, still server side quorum is maintained Client side quorum means, that the nodes are reachable by the client(IO path). If set to "auto", this option allows writes to the file only if number of bricks that are up >= ceil (of the total number of bricks that constitute that replica/2). If the number of replicas is even, then there is a further check: If the number of up bricks is exactly equal to n/2, then the first brick must be one of the bricks that is up. If it is more than n/2 then it is not necessary that the first brick is one of the up bricks. In x2 case, the first brick of a replica pair must be always up for data to be written from client. Hence when you bring down node1 you get readonly, but when you bring down node2 you can still write to the volume. ----- Original Message ----- From: "Atul Yadav" <atulyadavtech@xxxxxxxxx> To: "Atin Mukherjee" <amukherj@xxxxxxxxxx>, "gluster-users" <gluster-users@xxxxxxxxxxx> Sent: Monday, 14 November, 2016 8:04:24 PM Subject: Glusterfs readonly Issue Dear Team, In the event of the failure of master1, master 2 glusterfs home directory will become read only fs. If we manually shutdown the master 2, then there is no impact on the file system and all io operation will complete with out any problem. can you please provide some guidance to isolate the problem. # gluster peer status Number of Peers: 2 Hostname: master1-ib.dbt.au Uuid: a5608d66-a3c6-450e-a239-108668083ff2 State: Peer in Cluster (Connected) Hostname: compute01-ib.dbt.au Uuid: d2c47fc2-f673-4790-b368-d214a58c59f4 State: Peer in Cluster (Connected) # gluster vol info home Volume Name: home Type: Replicate Volume ID: 2403ddf9-c2e0-4930-bc94-734772ef099f Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp,rdma Bricks: Brick1: master1-ib.dbt.au:/glusterfs/home/brick1 Brick2: master2-ib.dbt.au:/glusterfs/home/brick2 Options Reconfigured: performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off network.remote-dio: enable cluster.quorum-type: auto nfs.disable: on performance.readdir-ahead: on cluster.server-quorum-type: server config.transport: tcp,rdma network.ping-timeout: 10 cluster.server-quorum-ratio: 51% cluster.enable-shared-storage: disable # gluster vol heal home info Brick master1-ib.dbt.au:/glusterfs/home/brick1 Status: Connected Number of entries: 0 Brick master2-ib.dbt.au:/glusterfs/home/brick2 Status: Connected Number of entries: 0 # gluster vol heal home info heal-failed Gathering list of heal failed entries on volume home has been unsuccessful on bricks that are down. Please check if all brick processes are running[root@master2 Thank You Atul Yadav _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
