We are running HTTPD as a proxy to a backend Tomcat application.
OS = RHEL 7.7
HTTPD:
Server version: Apache/2.4.34 (Red Hat)
Server built: Apr 3 2019 12:07:34
Server's Module Magic Number: 20120211:79
Server loaded: APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/opt/rh/httpd24/root/etc/httpd"
-D SUEXEC_BIN="/opt/rh/httpd24/root/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/opt/rh/httpd24/root/var/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
Tomcat:
Server version: Apache Tomcat/8.5.40
Server built: Apr 10 2019 14:31:19 UTC
Server number: 8.5.40.0
OS Name: Linux
OS Version: 3.10.0-693.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_162-b12
JVM Vendor: Oracle Corporation
What we are seeing on the proxy side are 2 'proxy errors' (scrubbed):
[Tue Jan 05 12:07:19.664861 2021] [proxy:error] [pid 74638:tid 140600384845568] [client
10.10.100.10:64959] AH00898: Error reading from remote server returned by /path/to/uri
[Tue Jan 05 12:07:19.664745 2021] [proxy_http:error] [pid 74638:tid 140600384845568] (70007)The timeout specified has expired: [client
10.10.100.10:64959] AH01102: error reading status line from remote server
node1.example.com:8002The client listed in these (10.10.100.10) is actually an A10 that is handling the VIP for the URL.
The configurations we have set on the proxy are:
<Proxy balancer://nodes>
BalancerMember
https://node1.example.com:8002 route=node1 keepalive=On ping=3 ttl=90 timeout=60
BalancerMember
https://node2.example.com:8002 route=node2 keepalive=On ping=3 ttl=90 timeout=60
ProxySet lbmethod=bybusyness
</Proxy>
On the Tomcat side we have:
<snip>
keepAliveTimeout="90000"
connectionTimeout="60000"
<snip>
My understanding is that the "ttl" on the HTTPD side corresponds to the "keepAliveTimeout" on the Tomcat side and the "timeout" on the HTTPD side corresponds to the "connectionTimeout" on the Tomcat side.
I am looking for some guidance into how we can successfully pinpoint where the issue lies. Is it the fact that the application is simply not responding to the request in the allotted configured settings? We can certainly test that option but being a PRD environment we'll need to schedule the process. Therefore, I'd like to get some thoughts on what we can do to pinpoint exactly what is going on here.
Thanks in advance,
HB
