Re: tests are timing out in master branch

[Amar Tumballi Suryanarayan <atumball@xxxxxxxxxx>]

Looks like after reverting a patch on RPC layer reconnection logic (https://review.gluster.org/22750) things are back to normal. 

For those who submitted a patch in last 1 week, please resubmit. (which should take care of rebasing on top of this patch).

This event proves that there are very delicate races in our RPC layer, which can trigger random failures. While it was discussed in brief earlier. We need to debug this further, and come up with possible next actions. Volunteers welcome.

I recommend to use https://github.com/gluster/glusterfs/issues/391 to capture our observations, and continue on github from here.

-Amar

On Wed, May 15, 2019 at 11:46 AM Sankarshan Mukhopadhyay <sankarshan.mukhopadhyay@xxxxxxxxx> wrote:

On Wed, May 15, 2019 at 11:24 AM Atin Mukherjee <amukherj@xxxxxxxxxx> wrote:
>
> There're random tests which are timing out after 200 secs. My belief is this is a major regression introduced by some commit recently or the builders have become extremely slow which I highly doubt. I'd request that we first figure out the cause, get master back to it's proper health and then get back to the review/merge queue.
>

For such dire situations, we also need to consider a proposal to back
out patches in order to keep the master healthy. The outcome we seek
is a healthy master - the isolation of the cause allows us to not
repeat the same offense.

> Sanju has already started looking into /tests/bugs/glusterd/optimized-basic-testcases-in-cluster.t to understand what test is specifically hanging and consuming more time.
_______________________________________________
Atin Mukherjee <amukherj@xxxxxxxxxx>, Sankarshan Mukhopadhyay <sankarshan.mukhopadhyay@xxxxxxxxx>
Community Meeting Calendar:

APAC Schedule -https://review.gluster.org/22750
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel

--

Amar Tumballi (amarts)

_______________________________________________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel