Anyone able to help us troubleshoot this issue? This is getting worse. We are back to our 3-replica setup but the issue is still happening. What we have found is that if I just bring one set of bricks offline. For example if I have (0 1 2) (3 4 5) (6 7 8) (9 10 11) and if I take the bricks 0 3 6 9, or bricks 1 4 7 10 offline then performance is super fast. The moment all bricks are online things become very slow. It seems like gluster is having some sort of lock contention between its members. During the period of slowness gluster profile would show excessive time spent in LOOKUP, FINODELK
11.60 752.64 us 10.00 us 2647757.00 us 272476323 LOOKUP
15.83 6884.12 us 29.00 us 2190470.00 us 40626259 WRITE
27.84 80480.22 us 40.00 us 11731910.00 us 6114072 FXATTROP
37.83 105125.18 us 12.00 us 276088722.00 us 6359515 FINODELK
We have about one or two months before we need to make a decision to keep Gluster and so far it has been a lot of headache.
On Thu, Jun 14, 2018 at 10:18 AM, Anh Vo <vtqanh@xxxxxxxxx> wrote:
Our gluster keeps getting to a state where it becomes painfully slow and many of our applications time out on read/write call. When this happens a simple ls at top level directory from the mount takes somewhere between 8-25s (normally it is very fast, at most 1-2s). The top level directory only has about 10 folders.The two methods to mitigate this problem have been 1) restart all GFS servers or 2) stop/start the volume. 2) does take somewhere between half an hour to an hour for gluster to get back to its desired performance.So far the logs don't show anything unusual but perhaps I don't know what I should be looking for in the logs. Even when gluster are fully functional we see lots of logs, hard to tell which error is harmless and what is not.This issue does not seem to happen with our 3 replica glusters, only with 2-replica-1-arbiter and 2-replica. However, our 3-replica glusters are only 30% full while the 2-replica ones are about 80% full.We're running 3.12.9 for the servers. The clients are 3.8.15, but we notice the slowness of operations on 3.12.9 clients as well.Configuration: 12 GFS servers, one brick per server, replica 2, 80T each brick. We used to have arbiters but thought the arbiters were causing the slow down so we took them out. Apparently it's not the case.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
