G'Day.
We've got a Subversion server (mod_dav_svn under Apache on Windows
Server 2003) set up behind another Apache server configured as a reverse
proxy (also on Windows Server 2003). We recently upgraded the
reverse-proxy box from Apache httpd 2.0.54 to httpd 2.2.11 and found
that when two users access the Subversion repository at the same time,
their usernames (or session tags maybe?) are being switched internally
by Apache. i.e. if the two users are simultaneously each trying to
access an area that the other doesn't have access to, they get back 401
Authorization Required errors, with the wrong usernames appearing in the
log (further explanation below). The Subversion server has been running
Apache 2.2.11 for months without trouble, its just the reverse-proxy
we've changed. We've isolated this error down to one of either
mod_proxy or mod_proxy_http (details below), so even though the problem
comes up when we use Subversion, with our custom authorization module,
we're pretty sure the bug lies in Apache.
At the moment we're fine sticking with Apache 2.0.63 (which happens to
be a lot faster than 2.2.6 as a reverse proxy, about 2x speed
difference), but if anyone can look into this issue and recommend
whether it's a reportable bug that'd be good. Thanks!
Example scenario of problem/symptoms:
- userA is doing a large (500KB, 30+ files) commit to
http://svnserver/repos/sandboxes/userA/private/
- userB is doing a large (500KB, 30+ files) commit to
http://svnserver/repos/sandboxes/userB/private/
- one user kicks off their commit while the other commit is already
running, i.e. in the middle of setting up the transaction & transferring
the files - or both are kicked off at the same time - doesn't matter
really, as long as both sessions are in progress simultaneously
- At that point, one OR both users will get a 401 error (there is no
particular pattern we've seen, other than that the user who kicks off
second is more likely to get the 401 while the other user's transaction
runs to completion, but about 1/3 of the time they both die)
- Looking at the access.log of the subversion server, it shows that (for
example) userB was denied access to userA's sandbox
(http://svnserver/repos/sandboxes/userA/private/fileXYZ), even though
userB was actually operating on their own sandbox at the time.
- Snippet of access.log pasted in at the bottom. We can reproduce this
fairly easily if additional Apache logging is necessary, but
unfortunately we can't supply any network-traffic dumps.
Details:
- Size of commit/transaction doesn't actually matter, its just that
longer transactions are more likely to collide in testing.
- Problem only occurs if accessing the svn repository via the reverse
proxy. i.e. Accessing the repository directly has no problems.
- Provided the commit is large enough that there is proper overlap, this
happens every time, its not intermittent. The only thing that changes
between tests is how many files the two commits get through before
dying, but it always dies in less than a second.
- No related messages in error.log (just the usual startup and shutdown
notices).
- Tested against different versions of Apache httpd and found some
interesting results:
Apache 2.0.54 - okay
Apache 2.0.63 - okay
Apache 2.2.2 - problem
Apache 2.2.3 - problem
Apache 2.2.4 - problem
Apache 2.2.6 - okay!!
Apache 2.2.8 - problem
Apache 2.2.9 - problem
Apache 2.2.10 - problem
Apache 2.2.11 - problem
Apache 2.2.11 with mod_proxy.so and mod_proxy_http.so copied in
from Apache 2.2.6 - okay!!
- i.e.the problem is in either mod_proxy.so or mod_proxy_http.so and
affects every release of Apache 2.2 except for 2.2.6!
- Authentication of users is via mod_auth_sspi, i.e. Windows Domain
authentication, at the Subversion server.
- Apache installation is from the binaries downloaded from
http://apache.16degrees.com.au/httpd/binaries/win32/ and
http://archive.apache.org/dist/httpd/binaries/win32/.
- Tried with and without the openssl build, no difference.
- For testing, both the subversion server and the reverse-proxy server
were put on the same LAN subnet so that test-clients could access
either/both (to confirm the problem only happened when the reverse proxy
was in the loop)
- Just for info completeness, we're running svn 1.5.4 but have also
checked svn 1.6.1: no difference in the problem.
- We'd been running the reverse-proxy with Apache 2.0.54 for over a year
with no trouble until the upgrade to 2.2.11; using the same httpd.conf
as below (except for the version-specific changes like ServerRoot and
mod_access vs mod_authz_host).
- Reverse-proxy to the svnserver is the only job the problematic Apache
installation is doing (I've listed below all of the modules it has
loaded).
Relevant conf lines from the reverse-proxy server:
--vv------<snip>---------vv--
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule setenvif_module modules/mod_setenvif.so
ProxyRequests off
ProxyVia On
<Location /repos >
ProxyPass http://svnserver.ourdomain/repos
ProxyPassReverse /repos/
ProxyPassReverse http://svnserver/repos
ProxyPassReverse http://svnserver.ourdomain/repos
<Limit OPTIONS PROPFIND GET REPORT MKACTIVITY PROPPATCH PUT
CHECKOUT MKCOL MOVE COPY DELETE LOCK UNLOCK MERGE>
Order Deny,Allow
Allow from all
Satisfy Any
</Limit>
</Location>
--^^------<snip>---------^^--
Relevant conf lines from subversion server ("svnserver") (note these are
not the only modules this server has loaded):
--vv------<snip>---------vv--
LoadModule sspi_auth_module modules/mod_auth_sspi.so
LoadModule dav_svn_module modules/mod_dav_svn.so
LoadModule permissions_svn_module modules/mod_permissions_svn.so
<Location /repos>
DAV svn
SVNPath //fileserver/svnrepos/repos
AuthType SSPI
AuthName "Subversion repository by OurDomain"
Require valid-user
SSPIAuth On
SSPIAuthoritative On
SSPIDomain OurDomain
SSPIOfferBasic On
PermissionsSVNAccessFile
//fileserver/svnrepos/conf/svs_permissions.conf
</Location>
--^^------<snip>---------^^--
mod_permissions_svn.so is an in-house simple-ish authorization module
based on mod_authz_svn that hooks into httpd core as follows:
--vv------<snip>---------vv--
static void registerHooks(apr_pool_t *p)
{
ap_hook_access_checker(checkAccess, NULL, NULL, APR_HOOK_LAST);
ap_hook_auth_checker(checkAuthorisation, NULL, NULL,
APR_HOOK_FIRST);
}
module AP_MODULE_DECLARE_DATA permissions_svn_module = {
STANDARD20_MODULE_STUFF,
createConfigRecord, /* config creater per dir */
NULL, /* dir merger --- default is to
override */
NULL, /* server config */
NULL, /* merge server config */
MODULE_ARGUMENTS, /* command apr_table_t */
registerHooks /* register hooks */
};
--^^------<snip>---------^^--
where checkAccess(...) and checkAuthorisation(...) implement our custom
rules using fields from the request_rec parameter such as user, uri,
etc. I only mention this to help convince anyone reading this that this
mod_permissions_svn is not the problem.
And finally a snippet of the access.log of the subversion server...
--vv------<snip>---------vv--
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000]
"PROPFIND
/repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugi
n.cpp HTTP/1.1" 207 776
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000]
"PROPFIND /repos/!svn/vcc/default HTTP/1.1" 207 455
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000]
"PROPFIND
/repos/!svn/bc/52575/sandboxes/fletched/private/tool/test/test_compatibi
lity/TestPlugin.cpp HTTP/1.1" 207 535
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000]
"CHECKOUT
/repos/!svn/ver/52575/sandboxes/fletched/private/tool/test/test_compatib
ility/TestPlugin.cpp HTTP/1.1" 201 307
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000] "PUT
/repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/p
rivate/testing_commit/tool/test/test_csvfile.cpp HTTP/1.1" 204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000] "PUT
/repos/!svn/wrk/11125f68-8c91-644c-85a7-b9a056d3b561/sandboxes/coopern/p
rivate/testing_commit/tool/tool_marshal_containers.h HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000]
"PROPFIND
/repos/sandboxes/fletched/private/tool/test/test_compatibility/TestPlugi
n.h HTTP/1.1" 401 401
xxx.xxx.xxx.198 - OURDOMAIN\\coopern [27/Apr/2009:14:36:02 +1000]
"DELETE /repos/!svn/act/10089f74-f41f-484e-b1a0-783dafcf2155 HTTP/1.1"
204 -
xxx.xxx.xxx.198 - OURDOMAIN\\fletched [27/Apr/2009:14:36:02 +1000]
"DELETE /repos/!svn/act/11125f68-8c91-644c-85a7-b9a056d3b561 HTTP/1.1"
204 -
--^^------<snip>---------^^--
Thanks!
Duncan.
IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the CRIMES ACT 1914. If you have received this email in error, you are requested to contact the sender and delete the email.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|