Hi,
We have an issue that I'd like to get some guidance on how to investigate further. We have a Tomcat application that is fronted by 3 HTTPD proxies (Apache/2.4.34) running mod_proxy_balancer. What we see in the HTTPD access logs are 400 response codes that include entries like:
preview.example.com 10.24.3.10 "-" - - [31/May/2022:15:16:30 -0700] "GET /BOTTOMS/shorts/c/0144 HTTP/1.1" 400 278 "
https://www.example.com/my-account/view" "Mozilla/5.0 (Linux; Android 12; SM-N975U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.78 Mobile Safari/537.36" "-" 293 9697 5006
preview.example.com 10.24.3.10 "-" - - [31/May/2022:15:35:13 -0700] "POST /checkout/multi/payment-method/add HTTP/1.1" 400 278 "
https://www.example.com/checkout/multi/payment-method/add" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" "-" 115 9435 0
/etc/httpd/conf/httpd.conf:# ------------------------------------------
# Listen Port
# ------------------------------------------
Listen
127.0.0.1:80Listen 443
# ------------------------------------------
# Load Modules
# ------------------------------------------
LoadModule ssl_module modules/mod_ssl.so
LoadModule systemd_module modules/mod_systemd.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule status_module modules/mod_status.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule alias_module modules/mod_alias.so
LoadModule dir_module modules/mod_dir.so
LoadModule mime_module modules/mod_mime.so
LoadModule proxy_hcheck_module modules/mod_proxy_hcheck.so
LoadModule watchdog_module modules/mod_watchdog.so
# ------------------------------------------
# Run As
# ------------------------------------------
User apache
Group apache
# ------------------------------------------
# Server Admin
# ------------------------------------------
ServerAdmin root@localhost
ServerTokens ProductOnly
Include conf.d/*.conf
# ------------------------------------------
# Doc Root
# ------------------------------------------
DocumentRoot /var/www/html
# ------------------------------------------
# Logs
# ------------------------------------------
ErrorLog "logs/error_log"
LogLevel warn
LogFormat "%v %h \"%{BALANCER_WORKER_NAME}e\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{JSESSIONID}C\" %D %I %O" combinedio
LogFormat "%v \"%{X-Forwarded-For}i\" \"%{BALANCER_WORKER_NAME}e\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{JSESSIONID}C\" %D %I %O" proxy
SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
CustomLog "logs/access_log" combinedio env=!forwarded
CustomLog "logs/access_log" proxy env=forwarded
# ------------------------------------------
# SSL
# ------------------------------------------
SSLSessionCache "shmcb:logs/session-cache(512000)"
SSLStaplingCache "shmcb:logs/stapling-cache(160000)"
# ------------------------------------------------
# Virtual Hosts
# ------------------------------------------------
<VirtualHost
127.0.0.1:80>
<Location "/serverstatus">
SetHandler server-status
</Location>
ErrorLog /dev/null
CustomLog /dev/null common
</VirtualHost>
/etc/httpd/conf.d/www.example.com.conf:<VirtualHost *:443>
ServerName
example.com ServerAlias
www.example.com SSLEngine on
SSLProxyEngine on
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLHonorCipherOrder On
SSLCompression off
SSLUseStapling on
SSLSessionTickets Off
SSLCertificateFile /etc/pki/tls/certs/file.crt
SSLCertificateKeyFile /etc/pki/tls/certs/file.key
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
Header always set X-Frame-Options SAMEORIGIN
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
Alias "/balancermanager_com" /var/www/html/balancermanager_com
<Location /balancermanager_com>
SetHandler balancer-manager
Order Deny,Allow
Deny from all
Allow from 10.1.1.56
</Location>
<Directory /var/www/html/maintenance>
Require all granted
</Directory>
ProxyHCExpr site_up {hc('body') !~ /ok/}
<Proxy balancer://storefront-com>
BalancerMember
https://app410.example.com:8443 route=app410 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app411.example.com:8443 route=app411 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app413.example.com:8443 route=app413 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app414.example.com:8443 route=app414 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app415.example.com:8443 route=app415 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app416.example.com:8443 route=app416 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app417.example.com:8443 route=app417 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app418.example.com:8443 route=app418 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
BalancerMember
https://app419.example.com:8443 route=app419 keepalive=On ttl=90 timeout=60 hcmethod=GET hcexpr=site_up hcuri=/healthcheck hcinterval=10 hcpasses=2 hcfails=2
ProxySet lbmethod=bybusyness
</Proxy>
RewriteEngine On
ErrorDocument 503 /maintenance/us/index.html
RewriteCond /var/www/html/maintenance/us/enabled -f
RewriteCond %{REQUEST_URI} !=/maintenance/us/index.html
RewriteRule ^ - [R=503,L]
RewriteCond /var/www/html/maintenance/us/enabled !-f
RewriteRule ^/maintenance/us/index.html$ / [R,L]
ProxyRequests Off
ProxyPreserveHost On
ProxyBadHeader Ignore
ProxyPassMatch .*\.php$ !
ProxyPassMatch .*\.asp$ !
ProxyPassMatch .*\.pl$ !
ProxyPassMatch .*\.pm$ !
ProxyPassMatch .*\.rb$ !
ProxyPassMatch .*\.py$ !
ProxyPass /maintenance !
Alias "/favicon.ico" /var/www/html/favicon.ico
ProxyPass "/" balancer://storefront-com/ stickysession=ROUTEID
ProxyPassReverse "/" balancer://storefront-com/ stickysession=ROUTEID
</VirtualHost>
What appears to be happening is that the requests are being 'addressed' by the alphabetically first *.conf file that is in /etc/httpd/conf.d. Previously we had another config file that alphabetically preceded the preview.example.com.conf config and the log entries referenced it as %v. But after turning the original alphabetically first config off, the log entries reference the now first
preview.example.com config. But as shown above, the referrer in all log entries is:
https://www.example.com.
Can anyone recommend how we can understand what might be the issue here?
Thanks in advance,
HB
