Load Balancing: 404 Not Found When Setting ProxyPass to Something Other Than "/"

[Michela <michela@xxxxxxxxxxxxxxxxxxxxxxxxxx>]

Hello, everyone!

I am using Apache as a load balancer, and have an issue with requests returning a 404 status code, unless they match a ProxyPass directive with a source value of "/". Can you help me identify the reason for this?

My load balancer configuration bits (just let me know if I should also send in the rest):

       RequestHeader set X-Forwarded-Proto https

       SSLProxyEngine On
       SSLProxyVerify None
       SSLProxyCheckPeerCN Off
       SSLProxyCheckPeerName Off
       SSLProxyCheckPeerExpire Off

       # IMPORTANT: Never remove ProxyRequests, and never set it to 'on'!
       ProxyPreserveHost On
       ProxyRequests off
       ProxyVia Off

       # Load balancer configuration
       ProxyHCExpr ok234 {%{REQUEST_STATUS} =~ /^[234]/}
       ProxyHCExpr gdown {%{REQUEST_STATUS} =~ /^[5]/}

       <Proxy balancer://static>
               BalancerMember http://127.0.0.1:8080
               ProxySet lbmethod=byrequests
       </Proxy>

       <Proxy balancer://dynamic>
               BalancerMember http://127.0.0.1:8080
               ProxySet lbmethod=byrequests
       </Proxy>

       ProxyPass        /assets balancer://static/
       ProxyPassReverse /assets balancer://static/

       ProxyPass        / balancer://dynamic/
       ProxyPassReverse / balancer://dynamic/

I've set the BalancerMember value for the "static" balancer to the same as the "dynamic" balancer not just for testing -- to eliminate my static asset cache server (Varnish) as a potential cause of this problem.

Another Apache virtual host is listening on 8080, which points to a Rails application running on Puma, served via a Unix socket. This is a staging server, the application servers in production will be on separate machines from the load balancers.

Any requests to this load balancer that start with '/assets' return 404 not found. If I comment out the ProxyPass and ProxyPassReverse lines for the '/assets' URI,  meaning all requests go to the same balancer ("dynamic"), then everything works great -- no 404s.

I have carefully reviewed the documentation at http://httpd.apache.org/docs/current/mod/mod_proxy_balancer.html several times, checked Apache's logs with debug verbosity, searched countless forums, mailing lists, blog posts, and the web in general for any indication of what I might have wrong here, and I still can't find the issue. That's strange to me, because I have seen several configuration examples online, where traffic from something other than "/" is sent to a given balancer member via ProxyPass, and I can't see anything significantly different between those (presumably properly working) configurations and my own.   :|

More information on my set-up:

Apache 2.4.37-39.0.1

OS: Enterprise Linux 8.4 (RHEL / Oracle / CentOS, etc)

Application: Custom Ruby on Rails application, running in Ruby 2.7, Rails 3.2, and Puma 5.4.0.

System is a virtual machine with two vCPUs and 4 GB RAM.

If I've missed any key information in my query, or if it would be helpful to have more information about the environment or problem, please tell me.   :)

Thank you for any help you may be able to provide!!

Best,

Michela