Re: Ceph Radosgw multi zone data replication failure

Vickey Singh <vickey.singh22693@xxxxxxxxx> · Tue, 28 Apr 2015 00:48:15 +0300

Hello Alfredo / Craig 
First of all Thank You So much for replying and giving your precious time to this problem. 

@Alfredo : I tried version radosgw-agent version 1.2.2 and the case has progressed a lot. ( below are some the logs )

I am now getting 

2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO  ] http://us-east-1.crosslogic.com:7480 endpoint does not support versioning
2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues reaching to endpoint http://us-east-1.crosslogic.com:7480
2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403: Forbidden

I am using CIVETWEB , any further help in this would be really helpful.

[root@us-east-1 ceph]#
[root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf
2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO  ]  __            __           __   ___      ___
2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO  ] /__` \ / |\ | /  `     /\  / _` |__  |\ |  |
2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ] .__/  |  | \| \__,    /~~\ \__> |___ | \|  |
2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ]                                      v1.2.2
2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ] agent options:
2015-04-28 00:35:14,752 5132 [radosgw_agent][INFO  ]  args:
2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]    conf                          : None
2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]    dest_access_key               : ****************
2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]    dest_secret_key               : ****************
2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]    destination                   : http://us-west-1.crosslogic.com:7480
2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]    incremental_sync_delay        : 30
2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO  ]    lock_timeout                  : 60
2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO  ]    log_file                      : /var/log/radosgw/radosgw-sync-us-east-west.log
2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO  ]    log_lock_time                 : 20
2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO  ]    max_entries                   : 1000
2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO  ]    metadata_only                 : False
2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO  ]    num_workers                   : 1
2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]    object_sync_timeout           : 216000
2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]    prepare_error_delay           : 10
2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]    quiet                         : False
2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]    rgw_data_log_window           : 30
2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]    source                        : None
2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]    src_access_key                : ****************
2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]    src_secret_key                : ****************
2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]    src_zone                      : None
2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]    sync_scope                    : incremental
2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO  ]    test_server_host              : None
2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO  ]    test_server_port              : 8080
2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO  ]    verbose                       : False
2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO  ]    versioned                     : False
2015-04-28 00:35:14,761 5132 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480
region map is: {u'us': [u'us-west', u'us-east']}
2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO  ] http://us-east-1.crosslogic.com:7480 endpoint does not support versioning
2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues reaching to endpoint http://us-east-1.crosslogic.com:7480
2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403: Forbidden
2015-04-28 00:35:14,782 5132 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://us-east-1.crosslogic.com:7480
2015-04-28 00:35:14,784 5132 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480
2015-04-28 00:35:14,785 5132 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://us-east-1.crosslogic.com:7480
2015-04-28 00:35:14,787 5132 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480
2015-04-28 00:35:14,807 5132 [radosgw_agent.sync][ERROR ] finding number of shards failed
2015-04-28 00:35:14,807 5132 [radosgw_agent.sync][WARNIN] error preparing for sync, will retry. Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 30, in prepare_sync
    syncer.prepare()
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 208, in prepare
    self.init_num_shards()
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 77, in init_num_shards
    self.num_shards = client.num_log_shards(self.src_conn, self.type)
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 403, in num_log_shards
    out = request(connection, 'get', 'admin/log', dict(type=shard_type))
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 147, in request
    check_result_status(result)
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 89, in check_result_status
    exc.HttpError)(result.status, result.reason)
HttpError: Http error code 403 content Forbidden
2015-04-28 00:35:24,826 5132 [radosgw_agent.sync][ERROR ] finding number of shards failed
2015-04-28 00:35:24,826 5132 [radosgw_agent.sync][WARNIN] error preparing for sync, will retry. Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 30, in prepare_sync
    syncer.prepare()
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 208, in prepare
    self.init_num_shards()
  File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 77, in init_num_shards
    self.num_shards = client.num_log_shards(self.src_conn, self.type)
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 403, in num_log_shards
    out = request(connection, 'get', 'admin/log', dict(type=shard_type))
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 147, in request
    check_result_status(result)
  File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 89, in check_result_status
    exc.HttpError)(result.status, result.reason)
HttpError: Http error code 403 content Forbidden
^C2015-04-28 00:35:25,609 5132 [radosgw_agent][ERROR ] KeyboardInterrupt

[root@us-east-1 ceph]#

[root@us-east-1 ceph]# curl  http://us-east-1.crosslogic.com:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[root@us-east-1 ceph]#
[root@us-east-1 ceph]#

[root@us-east-1 ceph]# curl  http://us-west-1.crosslogic.com:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
[root@us-east-1 ceph]#
[root@us-east-1 ceph]#

[root@us-east-1 ceph]# radosgw-admin region-map get --name client.radosgw.us-east-1
{ "regions": [

        { "key": "us",
          "val": { "name": "us",
              "api_name": "us",
              "is_master": "true",
              "endpoints": [
                    "http:\/\/us-east-1.crosslogic.com:7480\/"],
              "master_zone": "us-east",
              "zones": [
                    { "name": "us-east",
                      "endpoints": [
                            "http:\/\/us-east-1.crosslogic.com:7480\/"],
                      "log_meta": "true",
                      "log_data": "true"},
                    { "name": "us-west",
                      "endpoints": [
                            "http:\/\/us-west-1.crosslogic.com:7480\/"],
                      "log_meta": "true",
                      "log_data": "true"}],
              "placement_targets": [
                    { "name": "default-placement",
                      "tags": []}],
              "default_placement": "default-placement"}}],
  "master_region": "us",
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}[root@us-east-1 ceph]#
[root@us-east-1 ceph]#

[root@us-east-1 ceph]# radosgw-admin region-map get --name client.radosgw.us-west-1

{ "regions": [
        { "key": "us",
          "val": { "name": "us",
              "api_name": "us",
              "is_master": "true",
              "endpoints": [
                    "http:\/\/us-east-1.crosslogic.com:7480\/"],
              "master_zone": "us-east",
              "zones": [
                    { "name": "us-east",
                      "endpoints": [
                            "http:\/\/us-east-1.crosslogic.com:7480\/"],
                      "log_meta": "true",
                      "log_data": "true"},
                    { "name": "us-west",
                      "endpoints": [
                            "http:\/\/us-west-1.crosslogic.com:7480\/"],
                      "log_meta": "true",
                      "log_data": "true"}],
              "placement_targets": [
                    { "name": "default-placement",
                      "tags": []}],
              "default_placement": "default-placement"}}],
  "master_region": "us",
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1}}[root@us-east-1 ceph]#
[root@us-east-1 ceph]#

- VS -

On Mon, Apr 27, 2015 at 11:59 PM, Alfredo Deza <adeza@xxxxxxxxxx> wrote:
Hi Vickey (and all)

It looks like this issue was introduced as part of the 1.2.1 release.

I just finished getting 1.2.2 out (try upgrading please). You should no longer see that

error.

Hope that helps!

-Alfredo

----- Original Message -----

From: "Craig Lewis" <clewis@xxxxxxxxxxxxxxxxxx>

To: "Vickey Singh" <vickey.singh22693@xxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Sent: Monday, April 27, 2015 4:23:52 PM

Subject: Re:  Ceph Radosgw multi zone data replication failure

> [root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1

> [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

Are you trying to setup two zones on one cluster? That's possible, but you'll also want to spend some time on your CRUSH map making sure that the two zones are as independent as possible (no shared disks, etc).

Are you using Civetweb or Apache + FastCGI?

Can you include the output (from both clusters):

radosgw-admin --name=client.radosgw.us-east-1 region get

radosgw-admin --name=client.radosgw.us-east-1 zone get

Double check that both system users exist in both clusters, with the same secret.

On Sun, Apr 26, 2015 at 8:01 AM, Vickey Singh < vickey.singh22693@xxxxxxxxx > wrote:

Hello Geeks

I am trying to setup Ceph Radosgw multi site data replication using official documentation http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication

Everything seems to work except radosgw-agent sync , Request you to please check the below outputs and help me in any possible way.

Environment :

CentOS 7.0.1406

Ceph Versino 0.87.1

Rados Gateway configured using Civetweb

Radosgw zone list : Works nicely

[root@us-east-1 ceph]# radosgw-admin zone list --name client.radosgw.us-east-1

{ "zones": [

"us-west",

"us-east"]}

[root@us-east-1 ceph]#

Curl request to master zone : Works nicely

[root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480

<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" http://s3.amazonaws.com/doc/2006-03-01/ "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

[root@us-east-1 ceph]#

Curl request to secondary zone : Works nicely

[root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480

<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" http://s3.amazonaws.com/doc/2006-03-01/ "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

[root@us-east-1 ceph]#

Rados Gateway agent configuration file : Seems correct, no TYPO errors

[root@us-east-1 ceph] # cat cluster-data-sync.conf

src_access_key: M7QAKDH8CYGTK86CG93U

src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO

destination: http://us-west-1.crosslogic.com:7480

dest_access_key: ZQ32ES1WAWPG05YMZ7T7

dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM

log_file: /var/log/radosgw/radosgw-sync-us-east-west.log

[root@us-east-1 ceph]#

Rados Gateway agent SYNC : Fails , however it can fetch region map so i think src and dest KEYS are correct. But don't know why it fails on AttributeError

[root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf

region map is: {u'us': [u'us-west', u'us-east']}

Traceback (most recent call last):

File "/usr/bin/radosgw-agent", line 21, in <module>

sys.exit(main())

File "/usr/lib/python2.7/site-packages/radosgw_agent/cli.py", line 275, in main

except client.ClientException as e:

AttributeError: 'module' object has no attribute 'ClientException'

[root@us-east-1 ceph]#

Can query to Ceph cluster using us-east-1 ID

[root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

health HEALTH_OK

monmap e3: 3 mons at {ceph-node1= 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0 }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

osdmap e1063: 9 osds: 9 up, 9 in

pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#

Can query to Ceph cluster using us-west-1 ID

[root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

health HEALTH_OK

monmap e3: 3 mons at {ceph-node1= 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0 }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

osdmap e1063: 9 osds: 9 up, 9 in

pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#

Hope these packages are correct

[root@us-east-1 ceph]# rpm -qa | egrep -i "ceph|radosgw"

libcephfs1-0.87.1-0.el7.centos.x86_64

ceph-common-0.87.1-0.el7.centos.x86_64

python-ceph-0.87.1-0.el7.centos.x86_64

ceph-radosgw-0.87.1-0.el7.centos.x86_64

ceph-release-1-0.el7.noarch

ceph-0.87.1-0.el7.centos.x86_64

radosgw-agent-1.2.1-0.el7.centos.noarch

[root@us-east-1 ceph]#

Regards

VS

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com