Re: MPM event unstable behind AWS ALB (Apache Users)

I will still say this is a timeout issue. Set the ALB timeout to 3600 which is the max possible in case you do not know where to start from. I guess you already have ALB logs enabled.

In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout and make sure they make sense for your user case. Enable debug logs too for more details.

It is tough to guess without knowing your relevant configuration but 408 is usually caused by client connection being closed while keep-alive in use.

On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen <dan@xxxxxxxxxxxx> wrote:

Thanks for the response. Timeouts are configured appropriately...

To clarify; everything works fine through a TCP Network Load Balancer pointing at the same infrastructure. There is something about having a HTTP based Application Load Balancer in front of an MPM event configuration that's causing issues...

Dan
On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov <icicimov@xxxxxxxxx> wrote:
In a proxy chains like this getting the timeouts in sync is the most important thing. Make sure that you have done that.
On Mon, 20 Dec 2021, 08:37 Dan Washusen, <dan@xxxxxxxxxxxx> wrote:
Hi All,
I've been experimenting with the MPM event module with Apache instances sitting behind an AWS Application Load Balancer (ALB) and it really doesn't seem to be working well. Response times shoot up (compared to MPM event worker) and we see a fair few 502 errors returned (by the AWS ALB.

The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers

Debugging the issue I think I traced it down to Jetty returning a 408 error because it can't read the request body in a timely manner. So it seems like for some reason MPM is sending the request body...?

We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the following worker configuration:
ServerLimit 250
StartServers 100
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 64
MaxRequestWorkers 8000
I've come across several random posts mentioning that the MPM event module doesn't work behind an ALB but no-one seems to be into any detail. Anyone have some debugging/configuration suggestions?

Thanks,
Dan

p.s. I've created a serverfault post showing graphs etc: https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti