Re: Multisite Ceph setup sync issue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Amit,

 

Still same. Please see the below output. Anything else I can try? Please update

 

[cephuser@zabbix-client ~]$ radosgw-admin period update --commit 2>/dev/null

Sending period to new master zone 71931e0e-1be6-449f-af34-edb4166c4e4a

[cephuser@zabbix-client ~]$ sudo systemctl start ceph-radosgw@rgw.`hostname -s`

[cephuser@zabbix-client ~]$ sudo systemctl restart ceph-radosgw@rgw.`hostname -s`

[cephuser@zabbix-client ~]$ sudo systemctl status ceph-radosgw@rgw.`hostname -s`

ceph-radosgw@rgw.zabbix-client.service - Ceph rados gateway

   Loaded: loaded (/usr/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: disabled)

   Active: active (running) since Wed 2019-01-30 22:38:15 IST; 7s ago

Main PID: 8234 (radosgw)

   CGroup: /system.slice/system-ceph\x2dradosgw.slice/ceph-radosgw@rgw.zabbix-client.service

           └─8234 /usr/bin/radosgw -f --cluster ceph --name client.rgw.zabbix-client --setuser ceph --setgroup ceph

 

Jan 30 22:38:15 zabbix-client systemd[1]: Started Ceph rados gateway.

[cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone  noida1 2>/dev/null

          realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)

      zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)

           zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)

  metadata sync failed to read sync status: (2) No such file or directory

      data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)

                        failed to retrieve sync info: (5) Input/output error

[cephuser@zabbix-client ~]$

 

/Krishna

 

From: Amit Ghadge <amitg.b14@xxxxxxxxx>
Sent: Wednesday, January 30, 2019 9:17 PM
To: Krishna Verma <kverma@xxxxxxxxxxx>
Cc: Casey Bodley <cbodley@xxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Multisite Ceph setup sync issue

 

EXTERNAL MAIL

Have you commit your changes on slave gateway? 

First, run commit command on slave gateway and then try.

 

-AmitG

On Wed, 30 Jan 2019, 21:06 Krishna Verma, <kverma@xxxxxxxxxxx> wrote:

Hi Casey,

Thanks for your reply, however I tried with "--source-zone" option with sync command but getting below error:

Sync status From slave gateway to master zone "noida1"

[cephuser@zabbix-client ~]$ radosgw-admin sync status --source-zone  noida1 2>/dev/null
          realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
      zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
           zone 45c690a8-f39c-4b1d-9faf-e0e991ceaaac (san-jose)
  metadata sync failed to read sync status: (2) No such file or directory
      data sync source: 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
                        failed to retrieve sync info: (5) Input/output error
[cephuser@zabbix-client ~]$

Sync status From Master Gateway to slave zone " san-jose":

[cephuser@zabbix-server ~]$  radosgw-admin sync status --source-zone  san-jose 2>/dev/null
          realm 1102c891-d81c-480e-9487-c9f874287d13 (georep)
      zonegroup 74ad391b-fbca-4c05-b9e7-c90fd4851223 (noida)
           zone 71931e0e-1be6-449f-af34-edb4166c4e4a (noida1)
  metadata sync no sync (zone is master)
[cephuser@zabbix-server ~]$

Zone detail from master gateway :

[cephuser@zabbix-server ~]$ radosgw-admin zonegroup get  2>/dev/null
{
    "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
    "name": "noida",
    "api_name": "noida",
    "is_master": "true",
    "endpoints": [
        "http:\/\/zabbix-server:7480"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
    "zones": [
        {
            "id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
            "name": "noida1",
            "endpoints": [
                "http:\/\/vlno-ceph01:7480"
            ],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
}

[cephuser@zabbix-server ~]$


Zone detail from slave gateway:

[cephuser@zabbix-client ~]$ radosgw-admin zonegroup get  2>/dev/null
{
    "id": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
    "name": "noida",
    "api_name": "noida",
    "is_master": "true",
    "endpoints": [
        "http:\/\/zabbix-server:7480"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "71931e0e-1be6-449f-af34-edb4166c4e4a",
    "zones": [
        {
            "id": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac",
            "name": "san-jose",
            "endpoints": [
                "http:\/\/zabbix-client:7480"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        },
        {
            "id": "71931e0e-1be6-449f-af34-edb4166c4e4a",
            "name": "noida1",
            "endpoints": [
                "http:\/\/vlno-ceph01:7480"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "1102c891-d81c-480e-9487-c9f874287d13"
}

[cephuser@zabbix-client ~]

I need your expert advice.

/Krishna

-----Original Message-----
From: Casey Bodley <cbodley@xxxxxxxxxx>
Sent: Wednesday, January 30, 2019 1:54 AM
To: Krishna Verma <kverma@xxxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] Multisite Ceph setup sync issue

EXTERNAL MAIL


On Tue, Jan 29, 2019 at 12:24 PM Krishna Verma <kverma@xxxxxxxxxxx> wrote:
>
> Hi Ceph Users,
>
>
>
> I need your to fix sync issue in multisite setup.
>
>
>
> I have 2 cluster in different datacenter that we want to use for bidirectional data replication. By followed the documentation https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_docs_master_radosgw_multisite_&d=DwIFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=6MZxZ1DTyLWzj88W8kB9g8C3vLhZvcRI3-Xv_HdQ-Hg&s=Uw-eyENNATG6meKsTgmwdwMLYUD13mmDkpr9Eo2dqZo&e= I have setup the gateway on each site but when I am checking the sync status its getting failed as below:
>
>
>
> Admin node at master :
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin data sync status
>
> ERROR: source zone not specified
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin realm list
>
> {
>
>     "default_info": "1102c891-d81c-480e-9487-c9f874287d13",
>
>     "realms": [
>
>         "georep",
>
>         "geodata"
>
>     ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin zonegroup list
>
> read_default_id : 0
>
> {
>
>     "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
>
>     "zonegroups": [
>
>         "noida"
>
>     ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$ radosgw-admin zone list
>
> {
>
>     "default_info": "71931e0e-1be6-449f-af34-edb4166c4e4a",
>
>     "zones": [
>
>         "noida1"
>
>     ]
>
> }
>
>
>
> [cephuser@vlno-ceph01 cluster]$
>
>
>
> [cephuser@vlno-ceph01 cluster]$ cat ceph.conf
>
> [global]
>
> fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55
>
> mon_initial_members = vlno-ceph01
>
> mon_host = 172.23.16.67
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.23.16.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [cephuser@vlno-ceph01 cluster]$
>
>
>
> On Master Gateway :
>
>
>
> [cephuser@zabbix-server ~]$ cat /etc/ceph/ceph.conf
>
> [global]
>
> fsid = d52e50a4-ed2e-44cc-aa08-9309bc539a55
>
> mon_initial_members = vlno-ceph01
>
> mon_host = 172.23.16.67
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.23.16.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [client.rgw.zabbix-server]
>
> host = zabbix-server
>
> rgw frontends = "civetweb port=7480"
>
> rgw_zone=noida1
>
> [cephuser@zabbix-server ~]$
>
>
>
>
>
> On Secondary site admin node.
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin realm list
>
> {
>
>     "default_info": "1102c891-d81c-480e-9487-c9f874287d13",
>
>     "realms": [
>
>         "georep"
>
>     ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zonegroup list
>
> read_default_id : 0
>
> {
>
>     "default_info": "74ad391b-fbca-4c05-b9e7-c90fd4851223",
>
>     "zonegroups": [
>
>         "noida",
>
>         "default"
>
>     ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin zone list
>
> {
>
>     "default_info": "45c690a8-f39c-4b1d-9faf-e0e991ceaaac",
>
>     "zones": [
>
>         "san-jose"
>
>     ]
>
> }
>
>
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ cat ceph.conf
>
> [global]
>
> fsid = c626be3a-4536-48b9-8db8-470437052313
>
> mon_initial_members = vlsj-kverma1
>
> mon_host = 172.18.84.131
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.18.84.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
>
>
>
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
> [cephuser@vlsj-kverma1 cluster]$ radosgw-admin data sync status
>
> 2019-01-28 10:33:12.163298 7f11c24c79c0  1 Cannot find zone
> id=45c690a8-f39c-4b1d-9faf-e0e991ceaaac (name=san-jose), switching to
> local zonegroup configuration
>
> ERROR: source zone not specified
>
> [cephuser@vlsj-kverma1 cluster]$
>
>
>
> On Secondary site Gateway host:
>
>
>
> [cephuser@zabbix-client ceph]$ cat /etc/ceph/ceph.conf
>
> [global]
>
> fsid = c626be3a-4536-48b9-8db8-470437052313
>
> mon_initial_members = vlsj-kverma1
>
> mon_host = 172.18.84.131
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> # Your network address
>
> public network = 172.18.84.0/24
>
> osd pool default size = 2
>
> rgw_override_bucket_index_max_shards = 100
>
> debug ms = 1
>
> debug rgw = 20
>
> [client.rgw.zabbix-client]
>
> host = zabbix-client
>
> rgw frontends = "civetweb port=7480"
>
> rgw_zone=san-jose
>
>
>
> [cephuser@zabbix-client ceph]$
>
>
>
>
>
>
>
> Appreciate any help in the setup.
>
>
>
> /Krishna
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ceph.com_lis
> tinfo.cgi_ceph-2Dusers-2Dceph.com&d=DwIFaQ&c=aUq983L2pue2FqKFoP6PGHMJQ
> yoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=6MZ
> xZ1DTyLWzj88W8kB9g8C3vLhZvcRI3-Xv_HdQ-Hg&s=_22f69ZqTHaFPKMX21w1VTum2H_
> Nf-44JDmTqQOEH8o&e=

The 'radosgw-admin data sync status' command requires a --source-zone argument, which is generally the zone name on the opposite cluster.
But you're probably just looking for the 'radosgw-admin sync status'
command, which gives a summary of data and metadata sync against all zones. see https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_docs_master_radosgw_multisite_-23checking-2Dthe-2Dsync-2Dstatus&d=DwIFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=6MZxZ1DTyLWzj88W8kB9g8C3vLhZvcRI3-Xv_HdQ-Hg&s=3uXnPAw7BleYDrAf-QaRyxvPND_7HZnozAXclhTFbn4&e=
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux