Running Apache as a reverse proxy with long-lived client connections,
we're seeing memory consumption grow linearly in the number of
requests sent on the connection, to the tune of about 10 KB/request.
The consequence is that the memory footprint for each process is
determined by the longest-lived connection that process ever handles.
Our workaround is to lower MaxRequestsPerChild. Googling turns up a
blog entry that describes the same symptoms and workaround:
I've done some digging into the problem in hopes of finding a fix. I
haven't found one, but I wanted to share what I've learned and get
suggestions on next steps.
This is with Apache 2.2.6, and we observe similar behavior with Apache
2.0.54. We've also experimented with enabling/disabling keep-alive on
the downstream server, and see the same behavior.
I've run tests with between 50-1000 requests/connection and response
sizes of 10,000-100,000 bytes (the leak rate is independent of the
response size for this range).
All of the allocations are coming from the connection memory pool
(tagged "transaction").
According to valgrind+massif2, the problematic allocation point is
(line numbers are from 2.2.6):
apr_bucket_alloc (apr_buckets_alloc.c:142)
socket_bucket_read (apr_buckets_socket.c:34)
ap_core_input_filter (core_filters.c:245)
ap_get_brigade (util_filter.c:489)
ap_http_filter (http_filters.c:349)
ap_get_brigade (util_filter.c:489)
ap_proxy_http_process_response (mod_proxy_http.c:1494)
proxy_http_handler (mod_proxy_http.c:1742)
proxy_run_scheme_handler (mod_proxy.c:2155)
proxy_handler (mod_proxy.c:822)
ap_run_handler (config.c:158)
ap_invoke_handler (config.c:372)
If I understand apr_buckets_socket.c correctly, the bucket allocated
in socket_bucket_read is used to transfer data read from the
downstream server. It is not freed until the upstream connection
closes.
My best guess is that ap_proxy_http_process_response should register a
callback with the request pool to clean up the relevant data structure,
but I'm unclear on exactly what the call should be and when it should
be registered.
Any suggestions would be very welcome.
Thanks!
Ted
