Re: Load Balancing: 404 Not Found When Setting ProxyPass to Something Other Than "/"

[Michela <michela@xxxxxxxxxxxxxxxxxxxxxxxxxx>]

Hi, people. I figured out the problem in my load balancer.

Interestingly, when 'ProxyPass        /assets balancer://static/' is set, Apache, proxys the request on to the destination balancer member without the string that indicates the request to proxy (in this case, '/assets'). So if the request is 'http://domain.ca/assets/monalisa.jpg', the request passed on to the balancer member is actually 'http://domain.ca/monalisa.jpg'. Of course, the application server these requests were being sent to in my case does not have a file called "monalisa.jpg" (or the actual file in question, which of course, has a different name) at '/'. The file is at /assets/. So the application server returned a 404, like any good web application server should.

The solution then, was to add the string, '/assets' to the end of the target location in the ProxyPass directive.

So my ProxyPass directives look like this now:

       ProxyPass        /assets balancer://static/assets
       ProxyPassReverse /assets balancer://static/assets
 
       ProxyPass        / balancer://dynamic/
       ProxyPassReverse / balancer://dynamic/

Note the difference between the two conditions defined.

* The top ProxyPass directives send all requests starting with the string, '/assets' to 'balancer://static/assets', to the static asset cache (in my case), ensuring that the original URI of the request remains the same.

* The second set of ProxyPass and ProxyPassReverse directives send everything else to the other balancer.

Once I added that '/assets' bit to the end of those first ProxyPass directives, the requests for '/assets' all worked. No more 404s!

I've checked the responses coming from Apache and do see that those for '/assets' are sent on to the static asset cache as well -- the responses for them come back with Varnish headers.

So, hopefully this information is helpful to anyone else who wants to split requests from an Apache load balancer to two different load balance pools. I searched high and low and did not see any information about how to do this anywhere (though someone might have information posted about it somewhere). I will probably write a blog post about it myself as well.   :)

Best,

Michela

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, August 8th, 2021 at 5:15 PM, Michela <michela@xxxxxxxxxxxxxx> wrote:

Hello, everyone!

I am using Apache as a load balancer, and have an issue with requests returning a 404 status code, unless they match a ProxyPass directive with a source value of "/". Can you help me identify the reason for this?

My load balancer configuration bits (just let me know if I should also send in the rest):

       RequestHeader set X-Forwarded-Proto https

       SSLProxyEngine On
       SSLProxyVerify None
       SSLProxyCheckPeerCN Off
       SSLProxyCheckPeerName Off
       SSLProxyCheckPeerExpire Off

       # IMPORTANT: Never remove ProxyRequests, and never set it to 'on'!
       ProxyPreserveHost On
       ProxyRequests off
       ProxyVia Off

       # Load balancer configuration
       ProxyHCExpr ok234 {%{REQUEST_STATUS} =~ /^[234]/}
       ProxyHCExpr gdown {%{REQUEST_STATUS} =~ /^[5]/}

       <Proxy balancer://static>
               BalancerMember http://127.0.0.1:8080
               ProxySet lbmethod=byrequests
       </Proxy>

       <Proxy balancer://dynamic>
               BalancerMember http://127.0.0.1:8080
               ProxySet lbmethod=byrequests
       </Proxy>

       ProxyPass        /assets balancer://static/
       ProxyPassReverse /assets balancer://static/

       ProxyPass        / balancer://dynamic/
       ProxyPassReverse / balancer://dynamic/

I've set the BalancerMember value for the "static" balancer to the same as the "dynamic" balancer not just for testing -- to eliminate my static asset cache server (Varnish) as a potential cause of this problem.

Another Apache virtual host is listening on 8080, which points to a Rails application running on Puma, served via a Unix socket. This is a staging server, the application servers in production will be on separate machines from the load balancers.

Any requests to this load balancer that start with '/assets' return 404 not found. If I comment out the ProxyPass and ProxyPassReverse lines for the '/assets' URI,  meaning all requests go to the same balancer ("dynamic"), then everything works great -- no 404s.

I have carefully reviewed the documentation at http://httpd.apache.org/docs/current/mod/mod_proxy_balancer.html several times, checked Apache's logs with debug verbosity, searched countless forums, mailing lists, blog posts, and the web in general for any indication of what I might have wrong here, and I still can't find the issue. That's strange to me, because I have seen several configuration examples online, where traffic from something other than "/" is sent to a given balancer member via ProxyPass, and I can't see anything significantly different between those (presumably properly working) configurations and my own.   :|

More information on my set-up:

Apache 2.4.37-39.0.1

OS: Enterprise Linux 8.4 (RHEL / Oracle / CentOS, etc)

Application: Custom Ruby on Rails application, running in Ruby 2.7, Rails 3.2, and Puma 5.4.0.

System is a virtual machine with two vCPUs and 4 GB RAM.

If I've missed any key information in my query, or if it would be helpful to have more information about the environment or problem, please tell me.   :)

Thank you for any help you may be able to provide!!

Best,

Michela