This is a 18.2.4 reef cluster
Looking for help with cloud sync module. Here is my setup
1. Two Ceph clusters with RGW setup in a MultiSite configuration. There is no data synchronization between the clusters. No data sync groups, flows, or pipes.
2. We use a multi tenant setup with a small number of tenants (10). We have one tenant that is writing to the same bucket in different rgw zones. The names of files are unique in each zone.
3. We are using the cloud-sync module to copy data from our ceph clusters to aws. I have pasted the commands used to create the sync zone
The cloud-sync module works but for one issue that I can not figure out. For some reason another bucket in another tenant is showing up as target to sync but was never specified in the original config. I can never get a clean radosgw-admin sync status because there are recovering shards from a bucket in another tenant that I never intended to sync to the cloud.
The data sync status of the recovering shard is pointing to a bucket that never had any cloud sync specified. tenant1/bucket1 was never part of any tier config.
radosgw-admin sync error list --rgw-zone cloud-sync shows a never ending list of objects that can not be synced from tenant1/bucket1 there are 4 different types of error messages.
"message": "failed to sync bucket instance: (22) Invalid argument"
"message": "failed to sync bucket instance: (39) Directory not empty"
"message": "failed to sync object(16) Device or resource busy"
"message": "failed to sync object(39) Directory not empty"
One other point is that radosgw-admin sync info --rgw-zone cloud-sync --bucket tenant1/bucket1 seems to indicate that that is part of the
cloud-sync config.
What I just want to do is remove tenant1/bucket1 from the sync config and stop the data sync such that the shards recover and the error log stop
accumulating messages. I would really like a clean sync status for my cloud tier zone. It would also be nice to help understand what is happening here.
All and any help is greatly appreciated.
#
# sync and data sync status
#
radosgw-admin sync status --rgw-zone=cloud-sync
realm 02b82a81-7b16-4760-ab94-08b79176397e (realm)
zonegroup 0e44f33d-4e70-4ba6-8c75-794d3d0b69f8 (zonegroup)
zone 34e5298b-4537-4ada-b0d9-75579dc9640a (cloud-sync)
current time 2025-02-11T23:56:52Z
zonegroup features enabled: resharding
disabled: compress-encrypted
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 792edbc3-8516-4014-aad4-8e7b6ebe2757 (zone-1)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
1 shards are recovering
recovering shards: [72]
source: f3e72377-2fb1-4ab6-aa0a-76b91d157fb3 (zone-2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 4 shards
behind shards: [72,75,77,78]
oldest incremental change not applied: 2025-02-11T15:56:34.654956-0800 [75]
10 shards are recovering
recovering shards: [68,69,70,71,72,73,74,76,77,78]
radosgw-admin data sync status --rgw-zone cloud-sync --source-zone zone-2 --shard-id 68
{
"shard_id": 68,
"marker": {
"status": "incremental-sync",
"marker": "00000000000000000000:00000000000000357645",
"next_step_marker": "",
"total_entries": 1,
"pos": 0,
"timestamp": "2025-02-11T23:42:34.266680Z"
},
"pending_buckets": [
"tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0"
],
"recovering_buckets": [
"tenant1/bucket1:f3e72377-2fb1-4ab6-aa0a-76b91d157fb3.2251620.1:0[0]"
],
"current_time": "2025-02-11T23:58:08Z"
}
#
# cloud sync zone creation
#
radosgw-admin zone create \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--endpoints=https://ceph-server:4430 \
--tier-type=cloud
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=s-cloud-sync \
--access-key=<ACCESS> \
--secret=<SECRET>
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--tier-config=connections[].id=aws-cloud-sync
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--tier-config=connections[-1].endpoint=https://s3.amazonaws.com,connections[-1].access_key=<ACCESS>,connections[-1].secret=<SECRET>
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--tier-config=profiles[].connection_id=aws-cloud-sync
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--tier-config=profiles[-1].connection_id=aws-cloud-sync,profiles[-1].source_bucket='tenant0:bucket0',profiles[-1].target_path=aws-bucket
radosgw-admin zone modify \
--rgw-zonegroup=us \
--rgw-zone=cloud-sync \
--tier-config=connection_id=aws-cloud-sync,target_path=aws-bucket
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
