Re: [RGW] Setup 2 zones within a cluster does not sync data

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

according to [1] those are non-fatal errors:

These are the same non-fatal errors from above, where 404 errors from other zones are converted to -ENOENT. The RGWBackoffControlCR will continue to poll these objects for changes. These ERROR messages are unnecessarily spammy though, so I'd be in favor of removing them.

Can you show the sync status? Have you tried a full sync?

Regards,
Eugen

[1] https://www.spinics.net/lists/ceph-users/msg53477.html

Zitat von Huy Nguyen <viplanghe6@xxxxxxxxx>:

Hi,
Currently I'm testing replication ability between 2 zones in the same cluster. But only metadata is synced, not the data. I checked endpoints, system_key,... all good. If anyone have any idea, please guide me to resolve this situation. Thanks

The radosgw shows this log on both side, primary and secondary:

rgw rados thread: ERROR: failed to run sync
...
RGW-SYNC:data:sync:shard[54]: ERROR: failed to read remote data log info: ret=-5 RGW-SYNC:data:sync:shard[90]: ERROR: failed to read remote data log info: ret=-5
meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5
meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5

Here is my configure (it might be long):

zonegroup hn which contains hn-1 and hn-2, each zone have its own endpoint and radosgw instance on two different node:
{
    "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281",
    "epoch": 4,
    "predecessor_uuid": "e1981515-7355-4029-b26c-6be3de943be0",
    "sync_status": [],
    "period_map": {
        "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281",
        "zonegroups": [
            {
                "id": "9ced5b3b-5322-4ec9-862c-521bfff85c91",
                "name": "hn",
                "api_name": "hn",
                "is_master": true,
                "endpoints": [
                    "http://10.3.55.66:8080";
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf",
                "zones": [
                    {
                        "id": "0c5aac15-c899-46aa-bff9-353bf17f9602",
                        "name": "hn-2",
                        "endpoints": [
                            "http://10.3.53.107:8080";
                        ],
                        "log_meta": false,
                        "log_data": true,
                        "bucket_index_max_shards": 11,
                        "read_only": false,
                        "tier_type": "",
                        "sync_from_all": true,
                        "sync_from": [],
                        "redirect_zone": "",
                        "supported_features": [
                            "compress-encrypted",
                            "resharding"
                        ]
                    },
                    {
                        "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf",
                        "name": "hn-1",
                        "endpoints": [
                            "http://10.3.55.66:8080";
                        ],
                        "log_meta": false,
                        "log_data": true,
                        "bucket_index_max_shards": 11,
                        "read_only": false,
                        "tier_type": "",
                        "sync_from_all": true,
                        "sync_from": [],
                        "redirect_zone": "",
                        "supported_features": [
                            "compress-encrypted",
                            "resharding"
                        ]
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17",
                "sync_policy": {
                    "groups": []
                },
                "enabled_features": [
                    "resharding"
                ]
            }
        ],
        "short_zone_ids": [
            {
                "key": "0c5aac15-c899-46aa-bff9-353bf17f9602",
                "val": 3534260942
            },
            {
                "key": "2760e65c-b9d1-415a-929e-f5de2098d5cf",
                "val": 1656529937
            }
        ]
    },
    "master_zonegroup": "9ced5b3b-5322-4ec9-862c-521bfff85c91",
    "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
       ...ratelimit...
    },
    "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17",
    "realm_name": "ss-realm",
    "realm_epoch": 2
}

zone hn-1:

{
    "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf",
    "name": "hn-1",
    "domain_root": "hn-1.rgw.meta:root",
    "control_pool": "hn-1.rgw.control",
    "gc_pool": "hn-1.rgw.log:gc",
    "lc_pool": "hn-1.rgw.log:lc",
    "log_pool": "hn-1.rgw.log",
    "intent_log_pool": "hn-1.rgw.log:intent",
    "usage_log_pool": "hn-1.rgw.log:usage",
    "roles_pool": "hn-1.rgw.meta:roles",
    "reshard_pool": "hn-1.rgw.log:reshard",
    "user_keys_pool": "hn-1.rgw.meta:users.keys",
    "user_email_pool": "hn-1.rgw.meta:users.email",
    "user_swift_pool": "hn-1.rgw.meta:users.swift",
    "user_uid_pool": "hn-1.rgw.meta:users.uid",
    "otp_pool": "hn-1.rgw.otp",
    "system_key": {
        "access_key": "LIW8M2OTOCOHA41OJUGO",
        "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "hn-1.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "hn-1.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "hn-1.rgw.buckets.non-ec",
                "index_type": 0,
                "inline_data": true
            }
        }
    ],
    "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17",
    "notif_pool": "hn-1.rgw.log:notif"
}

zone hn-2:

{
    "id": "0c5aac15-c899-46aa-bff9-353bf17f9602",
    "name": "hn-2",
    "domain_root": "hn-2.rgw.meta:root",
    "control_pool": "hn-2.rgw.control",
    "gc_pool": "hn-2.rgw.log:gc",
    "lc_pool": "hn-2.rgw.log:lc",
    "log_pool": "hn-2.rgw.log",
    "intent_log_pool": "hn-2.rgw.log:intent",
    "usage_log_pool": "hn-2.rgw.log:usage",
    "roles_pool": "hn-2.rgw.meta:roles",
    "reshard_pool": "hn-2.rgw.log:reshard",
    "user_keys_pool": "hn-2.rgw.meta:users.keys",
    "user_email_pool": "hn-2.rgw.meta:users.email",
    "user_swift_pool": "hn-2.rgw.meta:users.swift",
    "user_uid_pool": "hn-2.rgw.meta:users.uid",
    "otp_pool": "hn-2.rgw.otp",
    "system_key": {
        "access_key": "LIW8M2OTOCOHA41OJUGO",
        "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "hn-2.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "hn-2.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "hn-2.rgw.buckets.non-ec",
                "index_type": 0,
                "inline_data": true
            }
        }
    ],
    "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17",
    "notif_pool": "hn-2.rgw.log:notif"
}
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux