Yeah, all those parameters are configured and everything works fine when I switch to the MPM worker module. The 408 errors are returned by Jetty and ProxyPass is configured to not reuse connections (which seems to suggest not a KeepAlive issue), Jetty is erroring because it receives no request data. Kind of seems like the MPM event module is missing events/data (when behind the ALB)...Given that the TCP network load balancer doesn't show the issue, maybe it's something to do with how the ALB sends the data to the instance (e.g. Jumbo frames vs the standard MTU)...?On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov <icicimov@xxxxxxxxx> wrote:I will still say this is a timeout issue. Set the ALB timeout to 3600 which is the max possible in case you do not know where to start from. I guess you already have ALB logs enabled.In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout and make sure they make sense for your user case. Enable debug logs too for more details.It is tough to guess without knowing your relevant configuration but 408 is usually caused by client connection being closed while keep-alive in use.On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen <dan@xxxxxxxxxxxx> wrote:Thanks for the response. Timeouts are configured appropriately...To clarify; everything works fine through a TCP Network Load Balancer pointing at the same infrastructure. There is something about having a HTTP based Application Load Balancer in front of an MPM event configuration that's causing issues...DanOn Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov <icicimov@xxxxxxxxx> wrote:In a proxy chains like this getting the timeouts in sync is the most important thing. Make sure that you have done that.On Mon, 20 Dec 2021, 08:37 Dan Washusen, <dan@xxxxxxxxxxxx> wrote:Hi All,I've been experimenting with the MPM event module with Apache instances sitting behind an AWS Application Load Balancer (ALB) and it really doesn't seem to be working well. Response times shoot up (compared to MPM event worker) and we see a fair few 502 errors returned (by the AWS ALB.The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App ServersDebugging the issue I think I traced it down to Jetty returning a 408 error because it can't read the request body in a timely manner. So it seems like for some reason MPM is sending the request body...?We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the following worker configuration:ServerLimit 250 StartServers 100 MinSpareThreads 75 MaxSpareThreads 250 ThreadLimit 64 ThreadsPerChild 64 MaxRequestWorkers 8000I've come across several random posts mentioning that the MPM event module doesn't work behind an ALB but no-one seems to be into any detail. Anyone have some debugging/configuration suggestions?Thanks,Danp.s. I've created a serverfault post showing graphs etc: https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti
|