Thanks! I'll continue with Apache 2.2 until the next release.
On Fri, Oct 24, 2014 at 8:58 AM, Yehuda Sadeh <yehuda@xxxxxxxxxx> wrote:
You're hitting issue #9206. Apache 2.4 filters out certain httpOn Thu, Oct 23, 2014 at 3:51 PM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote:
> I'm having a problem getting RadosGW replication to work after upgrading to
> Apache 2.4 on my primary test cluster. Upgrading the secondary cluster to
> Apache 2.4 doesn't cause any problems. Both Ceph's apache packages and
> Ubuntu's packages cause the same problem.
>
> I'm pretty sure I'm missing something obvious, but I'm not seeing it.
>
> Has anybody else upgraded their federated gateways to apache 2.4?
>
>
>
> My setup
> 2 VMs, each running their own ceph cluster with replication=1
> test0-ceph.cdlocal is the primary zone, named us-west
> test1-ceph.cdlocal is the secondary zone, named us-central
> Before I start, replication works, and I'm running
>
> Ubuntu 14.04 LTS
> Emperor (0.72.2-1precise, retained using apt-hold)
> Apache 2.2 (2.2.22-2precise.ceph, retained using apt-hold)
>
>
> As soon as I upgrade Apache to 2.4 in the primary cluster, replication gets
> permission errors. radosgw-agent.log:
> 2014-10-23T15:13:43.022 31106:ERROR:radosgw_agent.worker:failed to sync
> object bucket3/test6.jpg: state is error
>
> The access logs from the primary say (using vhost_combined log format):
> test0-ceph.cdlocal:80 172.16.205.1 - - [23/Oct/2014:15:16:51 -0700] "PUT
> /test6.jpg HTTP/1.1" 200 209 "-" "-"- - - [23/Oct/2014:13:24:18 -0700] "GET
> /?delimiter=/ HTTP/1.1" 200 1254 "-" "-" "bucket3.test0-ceph.cdlocal"
> <snip>
> test0-ceph.cdlocal:80 172.16.205.144 - - [23/Oct/2014:15:17:34 -0700] "GET
> /admin/log?marker=00000000089.89.3&type=bucket-index&bucket-instance=bucket3%3Aus-west.5697.2&max-entries=1000
> HTTP/1.1" 200 398 "-" "Boto/2.20.1 Python/2.7.6 Linux/3.13.0-37-generic"
> test0-ceph.cdlocal:80 172.16.205.144 - - [23/Oct/2014:15:17:34 -0700] "GET
> /bucket3/test6.jpg?rgwx-uid=us-central&rgwx-region=us&rgwx-prepend-metadata=us
> HTTP/1.1" 403 249 "-" "-"
>
> 172.16.205.143 is the primary cluster, .144 is the secondary cluster, and .1
> is my workstation.
>
>
> The access logs on the secondary show:
> test1-ceph.cdlocal:80 172.16.205.144 - - [23/Oct/2014:15:18:07 -0700] "GET
> /admin/replica_log?bounds&type=bucket-index&bucket-instance=bucket3%3Aus-west.5697.2
> HTTP/1.1" 200 643 "-" "Boto/2.20.1 Python/2.7.6 Linux/3.13.0-37-generic"
> test1-ceph.cdlocal:80 172.16.205.144 - - [23/Oct/2014:15:18:07 -0700] "PUT
> /bucket3/test6.jpg?rgwx-op-id=test1-ceph0.cdlocal%3A6484%3A3&rgwx-source-zone=us-west&rgwx-client-id=radosgw-agent
> HTTP/1.1" 403 286 "-" "Boto/2.20.1 Python/2.7.6 Linux/3.13.0-37-generic"
> test1-ceph.cdlocal:80 172.16.205.144 - - [23/Oct/2014:15:18:07 -0700] "GET
> /admin/opstate?client-id=radosgw-agent&object=bucket3%2Ftest6.jpg&op-id=test1-ceph0.cdlocal%3A6484%3A3
> HTTP/1.1" 200 355 "-" "Boto/2.20.1 Python/2.7.6 Linux/3.13.0-37-generic"
>
> If I crank up radosgw debugging, it tells me that the calculated digest is
> correct for the /admin/* requests, but fails for the object GET:
> /admin/log
> 2014-10-23 15:44:29.257688 7fa6fcfb9700 15 calculated
> digest=6Tt13P6naWJEc0mJmYyDj6NzBS8=
> 2014-10-23 15:44:29.257690 7fa6fcfb9700 15
> auth_sign=6Tt13P6naWJEc0mJmYyDj6NzBS8=
> /bucket3/test6.jpg
> 2014-10-23 15:44:29.411572 7fa6fc7b8700 15 calculated
> digest=pYWIOwRxCh4/bZ/D7b9RnS7RT1U=
> 2014-10-23 15:44:29.257691 7fa6fcfb9700 15 compare=0
> 2014-10-23 15:44:29.257693 7fa6fcfb9700 20 system request
> <snip>
> /bucket3/test6.jpg
> 2014-10-23 15:44:29.411572 7fa6fc7b8700 15 calculated
> digest=pYWIOwRxCh4/bZ/D7b9RnS7RT1U=
> 2014-10-23 15:44:29.411573 7fa6fc7b8700 15
> auth_sign=Gv398QNc6gLig9/0QbdO+1UZUq0=
> 2014-10-23 15:44:29.411574 7fa6fc7b8700 15 compare=-41
> 2014-10-23 15:44:29.411577 7fa6fc7b8700 10 failed to authorize request
>
> That explains the 403 responses.
>
> So I have metadata replication working, but the data replication is failing
> with permission problems. I verified that I can create users and buckets in
> the primary, and have them replicate to the secondary.
>
>
> A similar situation was posted to the list before. That time, the problem
> was that the system users weren't correctly deployed to both the primary and
> secondary clusters. I verified that both users exist in both clusters, with
> the same access and secret.
>
> Just to test, I used s3cmd. I can read and write to both clusters using
> both system user's credentials.
>
>
> Anybody have any ideas?
>
headers because they use underscores instead of dashes. There's a fix
for that for firefly, although it hasn't made it to an officially
released version.
Yehuda
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
